Time drift can prevent a Ceph cluster from starting
Bug #1519151 reported by
Gregory Elkinbard
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Triaged
|
Medium
|
MOS Ceph | ||
8.0.x |
Won't Fix
|
Medium
|
MOS Ceph | ||
Mitaka |
Triaged
|
Medium
|
MOS Ceph |
Bug Description
Clock skew of 0.05 sec can prevent Ceph cluster from starting for 20-30 minutes
While we do use NTP, unfortunately it is set to gradually adjust time and even a small clock drift will take a while to resolve.
Hardware in question was shutdown for 1 week to facilitate its transfer from one facility to another and accumulated a relatively trivial amount of clock drift, which prevented the Ceph cluster from starting up. Clock drift was so small that it was not visible using the date command.
We need to be more aggressive about syncing up the clocks on node start.
tags: | added: area-mos |
Changed in fuel: | |
assignee: | MOS Puppet Team (mos-puppet) → Stanislaw Bogatkin (sbogatkin) |
Changed in fuel: | |
assignee: | Herman Narkaytis (hnarkaytis) → MOS Ceph (mos-ceph) |
To post a comment you must log in.
It can be done by adding new option to puppet-ntp module. Like 'stepout' one, there is an option named 'step' which can regulate clock step offset. By set it value to <0.05, desired cluster state will be achieved much faster.