Cinder rbd errors connecting cluster during mon synchronization

[Artem Karamyshev]

Hi all. Today's we've got a issue. During long time synchronization, cinder-volume become unresponsible with rbd errors during connection to cluster. So we've got complete cinder down for 30 minute during synchronization.

Still the instances using ceph work good and we didn't lost the quorum for mons.

Log file in attach

[Gorka Eguileor (gorka)]

During the time that Cinder Volume was down, was the host where Cinder volume service was running able to access the Ceph cluster from the command line?

If it was, then this is probably related to the stats gathering confusion and performance issues we've been having [1][2][3][4] in the RBD driver that will very likely get fixed with the generic stats bug fix [5][6].

PS: By the logs I'm guessing this is the Ocata release.

[1] https://bugs.launchpad.net/cinder/+bug/1649956
[2] https://review.openstack.org/#/c/410884/
[3] https://bugs.launchpad.net/cinder/+bug/1704104
[4] https://review.openstack.org/#/c/483298/
[5] https://bugs.launchpad.net/cinder/+bug/1706060
[6] https://review.openstack.org/#/c/486734/

[Hanxi Liu (hanxi-liu)]

Gorka, I meet this same issue these days. The Ceph correlated volume service is down, but the Ceph cluster could be accessed using command lines and Ceph backend works well.