radosgw is down on one of the nodes

[Roman Podoliaka]

After a successful deployment one of the OSTF tests fails with the following error:

AssertionError: Step 2 failed: Dead backends ['object-storage node-3 Status: DOWN/L7STS Sessions: 0 Rate: 0 ']. Please refer to OpenStack logs for more details.

Apache can't connect to an FCGI service:

[Sun Feb 26 14:47:19.634837 2017] [proxy:error] [pid 31969:tid 140617598351104] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (127.0.0.1) failed
[Sun Feb 26 14:47:19.638199 2017] [proxy:error] [pid 31969:tid 140617598351104] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 60s
[Sun Feb 26 14:47:19.638235 2017] [proxy_fcgi:error] [pid 31969:tid 140617598351104] [client 240.0.0.2:41434] AH01079: failed to make connection to backend: 127.0.0.1
[Sun Feb 26 14:47:19.775148 2017] [proxy:error] [pid 31968:tid 140617615136512] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (127.0.0.1) failed
[Sun Feb 26 14:47:19.775275 2017] [proxy:error] [pid 31968:tid 140617615136512] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 60s
[Sun Feb 26 14:47:19.775288 2017] [proxy_fcgi:error] [pid 31968:tid 140617615136512] [client 10.109.8.3:42564] AH01079: failed to make connection to backend: 127.0.0.1
[Sun Feb 26 14:47:20.091005 2017] [proxy:error] [pid 31969:tid 140617556387584] AH00940: FCGI: disabled connection for (127.0.0.1)
[Sun Feb 26 14:47:21.192722 2017] [proxy:error] [pid 31969:tid 140617581565696] AH00940: FCGI: disabled connection for (127.0.0.1)

Apparently, it's due to the fact that something sends a SIGTERM to radosgw:

2017-02-26T14:47:17.789929+00:00 node-3 radosgw: 2017-02-26 14:47:17.789863 7fc424fa9700 1 ====== req done req=0x7fc478014b80 op status=0 http_status=200 ======
2017-02-26T14:47:18.133633+00:00 node-3 radosgw: 2017-02-26 14:47:18.127852 7fc4357ca700 1 ====== starting new request req=0x7fc47801adf0 =====
2017-02-26T14:47:18.137612+00:00 node-3 radosgw: 2017-02-26 14:47:18.135996 7fc4357ca700 1 ====== req done req=0x7fc47801adf0 op status=0 http_status=200 ======
2017-02-26T14:47:19.182115+00:00 node-3 radosgw: 2017-02-26 14:47:19.181887 7fc4407e0700 1 handle_sigterm
2017-02-26T14:47:19.182415+00:00 node-3 radosgw: 2017-02-26 14:47:19.181969 7fc4407e0700 1 handle_sigterm set alarm for 120
2017-02-26T14:47:19.183428+00:00 node-3 radosgw: 2017-02-26 14:47:19.183071 7fc4b6e02a00 -1 shutting down
2017-02-26T14:47:19.184979+00:00 node-3 radosgw: 2017-02-26 14:47:19.184220 7fc4407e0700 1 handle_sigterm
2017-02-26T14:47:19.185190+00:00 node-3 radosgw: 2017-02-26 14:47:19.184712 7fc43ffdf700 0 ERROR: FCGX_Accept_r returned -4
2017-02-26T14:47:19.306435+00:00 node-3 radosgw: 2017-02-26 14:47:19.306298 7fc4b6e02a00 1 final shutdown

https://packaging-ci.infra.mirantis.net/job/master-pkg-mos-systest-ubuntu-xenial/343/