Time drift can prevent a Ceph cluster from starting

[Gregory Elkinbard]

Clock skew of 0.05 sec can prevent Ceph cluster from starting for 20-30 minutes
While we do use NTP, unfortunately it is set to gradually adjust time and even a small clock drift will take a while to resolve.
Hardware in question was shutdown for 1 week to facilitate its transfer from one facility to another and accumulated a relatively trivial amount of clock drift, which prevented the Ceph cluster from starting up. Clock drift was so small that it was not visible using the date command.

We need to be more aggressive about syncing up the clocks on node start.

[Stanislaw Bogatkin (sbogatkin)]

It can be done by adding new option to puppet-ntp module. Like 'stepout' one, there is an option named 'step' which can regulate clock step offset. By set it value to <0.05, desired cluster state will be achieved much faster.

[Stanislaw Bogatkin (sbogatkin)]

Also, I have a question to ceph guys - maybe it is better to fix ceph tolerancy to time variance? We have tons of other soft (and cluster one, like corosync) which have much better tolerancy. Why ceph needs 50ms? Can this time be bigger, for example 150ms?

[Roman Podoliaka (rpodolyaka)]

We no longer fix Medium bugs in 8.0, closing as Won't Fix