Ceph driver should support retries on rados_connect_timeout

[Ivan Kolodyazhny]

RBD driver supports rados_connect_timeout param for connection to rados. It would be good to have a retries support if connection timeout occues

[Josh Durgin (jdurgin)]

librados already retries internally. Doing it at the cinder layer when timeouts are enabled is equivalent to using a longer timeout. It'd be better imo to make the rbd driver work as you like even when timeouts are disabled.

[Ivan Kolodyazhny (e0ne)]

Josh, thanks for a comment about librados.

"librados already retries internally. Doing it at the cinder layer when timeouts are enabled is equivalent to using a longer timeout"

Unfortunately, I didn't find anything how librados works with connection timeouts and retries. Could you please share a link with it?
I don't know C++ well so it's pretty hard for me to find it in the sources.

"It'd be better imo to make the rbd driver work as you like even when timeouts are disabled."
What do you mean? How should we handle case when connection to 'hangs' or something like it happened?

[Josh Durgin (jdurgin)]

There aren't good docs for this behavior of librados. Basically anything to do with the cluster may block and retry internally, to hide the complexity of retrying everything from library users. This could happen for many reasons. Network connections can flap. Switches can stop working. Daemons can be restarted. When a disk fails, an I/O needs to be resent to a different osd. An object can be unavailable for writes if not enough copies of it are available. The cluster could be temporarily full. In each of these cases (and many more), librados retries for the library user. Higher-level timeouts in things on top of rados are possible, and this eliminates extra configuration to make any timeouts at the librados level match higher levels.

The timeout options for librados were added for library users to simplify use cases where they want to fail eventually, and not block for the cluster to become available. It makes sense to timeout in cases like updating storage usage, which don't need to be strictly up to date, and will be re-run by a higher level (in this case a periodic task). I'm not sure what other cases would make sense in cinder off the top of my head. Config issues like a missing or incorrect ceph key tend to be immediate failures anyway.

Did you have particular things in mind for retrying, or a particular failure condition where connect() wouldn't eventually work?

[Ivan Kolodyazhny (e0ne)]

John,
I'm hitting with issue when connection to Ceph hangs. I don't have clear steps to reproduce so I didn't file a bug yet. You can find some more details here: https://bugs.launchpad.net/mos/6.1.x/+bug/1459781. I'll post bug to Cinder once I'll get more details

[Josh Durgin (jdurgin)]

It's not clear it's hanging trying to connect or at a later step. I'd suggest enabling ceph logging for cinder-volume (log file = /var/log/cinder/rbd.$pid.$cctid.log, debug ms = 1, debug rbd = 20 in the [client] section of ceph.conf).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/190579