RadosGW process dies on cluster restart

[Gregory Elkinbard]

There is a start up dependency between Ceph OSDs, Mons and RadosGW.
If RadosGW is started before OSDs and Mons are online, it will exit with an error.
This creates an unnecessary complication for the cluster start up process.

RadosGW should be registered with Supervisor to restart the process if it exits and correct pacemaker dependencies should be created

[Vladimir Kuklin (vkuklin)]

So after discussion with bug reporter, this is related to cluster cold boot-up process and requires radosgw to be monitored and restarted properly. It could be useful to use respawn upstart configuration or systemd control in the future. Nevertheless, the proper fix is to make radosgw retry connection until it finds running osds and can start providing the service. I am marking this bug as high and targeting assigning it to our ceph team for investigation

[Herman Narkaytis (hnarkaytis)]

This kind of the change in RGW will have resistance from the community and core team. It will take at least one release cycle to push change upstream and another cycle to downstream it to MOS. I.e. we will have this fix in a year from now.

At my understanding retry should be implemented on a level of startup scripts.

[Andrey Maximov (maximov)]

so my understanding that we can fix this problem either by fixing RadowGW (good but long term solution) or by providing workaround in the deployment scripts for RadosGW.
So moving to CEPH team as deployment manifests for CEPH is currently CEPH team responsibility.

[Herman Narkaytis (hnarkaytis)]

"no longer affects: mos/8.0.x" - means that we do not expect any kind of solution for 8.0?

[Radoslaw Zarzynski (rzarzynski)]

radosgw already has a mechanism for waiting for monitors and OSDs.
It works quite well at least on a my development cluster:

/work/ceph-v0.80.9/src$ pgrep ceph-osd
/work/ceph-v0.80.9/src$ pgrep ceph-mon
/work/ceph-v0.80.9/src$ ./radosgw -n client.radosgw.gateway -c ./ceph-rgw.conf -d
2015-11-30 14:16:51.912805 7efd80a2a7c0 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process lt-radosgw, pid 17886
2015-11-30 14:16:51.916629 7efd80a22700 0 -- :/1017886 >> 127.0.0.1:6789/0 pipe(0x8dd650 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x8dd8e0).fault
2015-11-30 14:16:54.917334 7efd74bdc700 0 -- :/1017886 >> 127.0.0.1:6791/0 pipe(0x7efd60000c00 sd=8 :0 s=1 pgs=0 cs=0 l=1 c=0x7efd60000e90).fault
2015-11-30 14:16:57.918040 7efd80a22700 0 -- :/1017886 >> 127.0.0.1:6790/0 pipe(0x7efd60003010 sd=8 :0 s=1 pgs=0 cs=0 l=1 c=0x7efd600032a0).fault

After brining monitors and OSDs the radosgw instance became fully
functional. It has also survived restart of all OSD and mons in the cluster.

[Alexei Sheplyakov (asheplyakov)]

tl;dr: as of Ceph 0.94.5 (hammer) radosgw automatically reconnects to monitors and OSDs.
No special options (command line switches) are necessary (reconnection works out of the box).
If no monitors can be found within a certain time (~ several minutes) radosgw bails out (and get
restarted by upstart/systemd). Perhaps the problem occurs only under a special conditions.
Please specify the exact steps describe the steps to reproduce the bug so we can reproduce and fix it.

The test environment
--------------------

1 monitor (saceph-mon)
3 OSDs (saceph-osd1, saceph-osd2, saceph-osd3)
1 radosgw node (saceph-rgw)

Ceph configuration details
---------------------------

# /etc/ceph/ceph.conf
[global]
fsid = 17875282-4597-4e4e-805a-a69919dbeb0c
mon_initial_members = saceph-mon
mon_host = 10.253.0.20
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

[client.radosgw.gateway]
host = saceph-rgw
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = ""
log file = /var/log/radosgw/client.radosgw.gateway.log
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0
rgw print continue = false

# /etc/apache2/conf-enabled/rgw.conf

<VirtualHost *:80>
ServerName localhost
DocumentRoot /var/www/html

ErrorLog /var/log/apache2/rgw_error.log
CustomLog /var/log/apache2/rgw_access.log combined

RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

SetEnv proxy-nokeepalive 1
ProxyPass / fcgi://localhost:9000/

</VirtualHost>

Tests
=====

Restarting monitor
-------------------

ssh saceph-mon sh -c "'service ceph-mon-all stop && sleep 60 && service ceph-mon-all start'"

radosgw notes that monitor is not available any more and tries to reconnect:

2015-12-31 12:58:27.478011 7f47ed54d700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c400a050 sd=8 :46741 s=2 pgs=2 cs=1 l=1 c=0x7f47c40068e0).reader couldn
't read tag, (0) Success
2015-12-31 12:58:27.478054 7f47ed54d700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c400a050 sd=8 :46741 s=2 pgs=2 cs=1 l=1 c=0x7f47c40068e0).fault (0) Suc
cess
2015-12-31 12:58:27.479242 7f47e114c700 0 monclient: hunting for new mon
2015-12-31 12:58:27.479245 7f47e114c700 1 -- 10.253.0.254:0/1015759 mark_down 0x7f47c40068e0 -- pipe dne
2015-12-31 12:58:27.479287 7f47e114c700 1 -- 10.253.0.254:0/1015759 --> 10.253.0.20:6789/0 -- auth(proto 0 40 bytes epoch 1) v1 -- ?+0 0x7f47c4014cf0 con 0x7f47c4013b70
2015-12-31 12:58:27.479503 7f47edbc4700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c4015090 sd=26 :0 s=1 pgs=0 cs=0 l=1 c=0x7f47c4013b70).connect error 10.253.0.20:6789/0, (111) Connection refused
2015-12-31 12:58:27.479550 7f47edbc4700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c4015090 sd=26 :0 s=1 pgs=0 cs=0 l=1 c=0x7f47c4013b70).fault (111) Connection refused

The connection gets restored after the monitor has been restarted:

2015-12-31 12:59:34.864740 7f47e114c700 1 -- 10.253.0.254:0/1015759 <== mon.0 10.253.0.20:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3166451580 0 0) 0x7f47d0003dc0 con 0x7f47b401e540
2015-12-31 12:59:34.865041...

[Roman Podoliaka (rpodolyaka)]

Greg, please see the Alexei's comment ^

[Roman Podoliaka (rpodolyaka)]

We believe, this works in 8.0. Please re-open the bug, if you disagree.

[Dina Belova (dbelova)]

No more information provided, moving to invalid after 1 month in Incomplete state.