RadosGW process dies on cluster restart

Bug #1519148 reported by Gregory Elkinbard
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Gregory Elkinbard
7.0.x
Invalid
High
Gregory Elkinbard

Bug Description

There is a start up dependency between Ceph OSDs, Mons and RadosGW.
If RadosGW is started before OSDs and Mons are online, it will exit with an error.
This creates an unnecessary complication for the cluster start up process.

RadosGW should be registered with Supervisor to restart the process if it exits and correct pacemaker dependencies should be created

Tags: area-ceph
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
tags: added: area-library
Changed in fuel:
milestone: none → 8.0
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

So after discussion with bug reporter, this is related to cluster cold boot-up process and requires radosgw to be monitored and restarted properly. It could be useful to use respawn upstart configuration or systemd control in the future. Nevertheless, the proper fix is to make radosgw retry connection until it finds running osds and can start providing the service. I am marking this bug as high and targeting assigning it to our ceph team for investigation

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → MOS Ceph (mos-ceph)
assignee: MOS Ceph (mos-ceph) → Fuel Library Team (fuel-library)
Changed in mos:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → MOS Ceph (mos-ceph)
milestone: none → 8.0
Revision history for this message
Herman Narkaytis (hnarkaytis) wrote :

This kind of the change in RGW will have resistance from the community and core team. It will take at least one release cycle to push change upstream and another cycle to downstream it to MOS. I.e. we will have this fix in a year from now.

At my understanding retry should be implemented on a level of startup scripts.

Revision history for this message
Andrey Maximov (maximov) wrote :

so my understanding that we can fix this problem either by fixing RadowGW (good but long term solution) or by providing workaround in the deployment scripts for RadosGW.
So moving to CEPH team as deployment manifests for CEPH is currently CEPH team responsibility.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → MOS Ceph (mos-ceph)
tags: added: area-mos
removed: area-library
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel
Dmitry Pyzhov (dpyzhov)
no longer affects: mos/8.0.x
Revision history for this message
Herman Narkaytis (hnarkaytis) wrote :

"no longer affects: mos/8.0.x" - means that we do not expect any kind of solution for 8.0?

Changed in mos:
assignee: Radoslaw Zarzynski (rzarzynski) → Kostiantyn Danylov (kdanylov)
Revision history for this message
Radoslaw Zarzynski (rzarzynski) wrote :

radosgw already has a mechanism for waiting for monitors and OSDs.
It works quite well at least on a my development cluster:

/work/ceph-v0.80.9/src$ pgrep ceph-osd
/work/ceph-v0.80.9/src$ pgrep ceph-mon
/work/ceph-v0.80.9/src$ ./radosgw -n client.radosgw.gateway -c ./ceph-rgw.conf -d
2015-11-30 14:16:51.912805 7efd80a2a7c0 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process lt-radosgw, pid 17886
2015-11-30 14:16:51.916629 7efd80a22700 0 -- :/1017886 >> 127.0.0.1:6789/0 pipe(0x8dd650 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x8dd8e0).fault
2015-11-30 14:16:54.917334 7efd74bdc700 0 -- :/1017886 >> 127.0.0.1:6791/0 pipe(0x7efd60000c00 sd=8 :0 s=1 pgs=0 cs=0 l=1 c=0x7efd60000e90).fault
2015-11-30 14:16:57.918040 7efd80a22700 0 -- :/1017886 >> 127.0.0.1:6790/0 pipe(0x7efd60003010 sd=8 :0 s=1 pgs=0 cs=0 l=1 c=0x7efd600032a0).fault

After brining monitors and OSDs the radosgw instance became fully
functional. It has also survived restart of all OSD and mons in the cluster.

Changed in mos:
assignee: Kostiantyn Danylov (kdanylov) → Alexei Sheplyakov (asheplyakov)
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :
Download full text (5.5 KiB)

tl;dr: as of Ceph 0.94.5 (hammer) radosgw automatically reconnects to monitors and OSDs.
No special options (command line switches) are necessary (reconnection works out of the box).
If no monitors can be found within a certain time (~ several minutes) radosgw bails out (and get
restarted by upstart/systemd). Perhaps the problem occurs only under a special conditions.
Please specify the exact steps describe the steps to reproduce the bug so we can reproduce and fix it.

The test environment
--------------------

1 monitor (saceph-mon)
3 OSDs (saceph-osd1, saceph-osd2, saceph-osd3)
1 radosgw node (saceph-rgw)

Ceph configuration details
---------------------------

# /etc/ceph/ceph.conf
[global]
fsid = 17875282-4597-4e4e-805a-a69919dbeb0c
mon_initial_members = saceph-mon
mon_host = 10.253.0.20
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

[client.radosgw.gateway]
host = saceph-rgw
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = ""
log file = /var/log/radosgw/client.radosgw.gateway.log
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0
rgw print continue = false

# /etc/apache2/conf-enabled/rgw.conf

<VirtualHost *:80>
ServerName localhost
DocumentRoot /var/www/html

ErrorLog /var/log/apache2/rgw_error.log
CustomLog /var/log/apache2/rgw_access.log combined

RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

SetEnv proxy-nokeepalive 1
ProxyPass / fcgi://localhost:9000/

</VirtualHost>

Tests
=====

Restarting monitor
-------------------

ssh saceph-mon sh -c "'service ceph-mon-all stop && sleep 60 && service ceph-mon-all start'"

radosgw notes that monitor is not available any more and tries to reconnect:

2015-12-31 12:58:27.478011 7f47ed54d700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c400a050 sd=8 :46741 s=2 pgs=2 cs=1 l=1 c=0x7f47c40068e0).reader couldn
't read tag, (0) Success
2015-12-31 12:58:27.478054 7f47ed54d700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c400a050 sd=8 :46741 s=2 pgs=2 cs=1 l=1 c=0x7f47c40068e0).fault (0) Suc
cess
2015-12-31 12:58:27.479242 7f47e114c700 0 monclient: hunting for new mon
2015-12-31 12:58:27.479245 7f47e114c700 1 -- 10.253.0.254:0/1015759 mark_down 0x7f47c40068e0 -- pipe dne
2015-12-31 12:58:27.479287 7f47e114c700 1 -- 10.253.0.254:0/1015759 --> 10.253.0.20:6789/0 -- auth(proto 0 40 bytes epoch 1) v1 -- ?+0 0x7f47c4014cf0 con 0x7f47c4013b70
2015-12-31 12:58:27.479503 7f47edbc4700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c4015090 sd=26 :0 s=1 pgs=0 cs=0 l=1 c=0x7f47c4013b70).connect error 10.253.0.20:6789/0, (111) Connection refused
2015-12-31 12:58:27.479550 7f47edbc4700 2 -- 10.253.0.254:0/1015759 >> 10.253.0.20:6789/0 pipe(0x7f47c4015090 sd=26 :0 s=1 pgs=0 cs=0 l=1 c=0x7f47c4013b70).fault (111) Connection refused

The connection gets restored after the monitor has been restarted:

2015-12-31 12:59:34.864740 7f47e114c700 1 -- 10.253.0.254:0/1015759 <== mon.0 10.253.0.20:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3166451580 0 0) 0x7f47d0003dc0 con 0x7f47b401e540
2015-12-31 12:59:34.865041...

Read more...

Changed in mos:
status: Confirmed → Incomplete
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Greg, please see the Alexei's comment ^

tags: added: area-ceph
removed: area-mos
Changed in mos:
assignee: Alexei Sheplyakov (asheplyakov) → Gregory Elkinbard (gelkinbard)
Changed in mos:
status: Incomplete → Confirmed
status: Confirmed → Incomplete
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

We believe, this works in 8.0. Please re-open the bug, if you disagree.

Changed in mos:
status: Incomplete → Invalid
Revision history for this message
Dina Belova (dbelova) wrote :

No more information provided, moving to invalid after 1 month in Incomplete state.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.