Hammer -> Jewel resulted in osd re-init + rocksdb corruption
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceph OSD Charm |
Confirmed
|
High
|
Unassigned |
Bug Description
Ceph uses a 'ready' touchfile to indicate that an OSD has already been
initialized by mkfs [0]. The ceph-osd charm overloads the 'ready' touchfile to
also indicate whether an OSD is enabled/disabled for upstart [1].
Ceph's upstart config prevents ceph-osd from being started when 'ready' doesn't
exit. However, it does not prevent calls to ceph-disk [2].
During upgrade from Hammer -> Jewel, encountered a scenario where ceph-disk was
started while the OSD 'ready' touchfile was removed from an existing OSD. This
resulted in re-initialization of the OSD. Normally re-init would be benign, but
on Jewel 10.2.11 new OSDs are rocksdb by default [3]. The OSD re-init behavior
corrupted the existing leveldb by partially converting it to rocksdb.
Recommend modifying the charm to use an upstart method to police the service.
For example, this would disable a service:
`echo manual | sudo tee /etc/init/
This will let ceph manage the ready touchfile and keep it tightly bound to
whether the OSD should mkfs.
[0] https:/
[1] https:/
[2] https:/
[3] https:/
Changed in charm-ceph-osd: | |
status: | New → Confirmed |
importance: | Undecided → High |
tags: | added: ceph-upgrade |
Discussed the proposed solution to use upstart's service 'override' with Felipe.
Unfortunately, this would not allow granular control of which OSDs get enabled/disabled.
Stepping back and looking at the code flow in this scenario: The only reason the
charm needs granular control to disable an OSD is because file ownership changes
may take a significantly long time to complete within the osd directory.
Some thoughts here:
* Why does the OSD need to be stopped while ownership updates are completed?
* Why aren't the OSDs disabled prior to updating code?
It seems like the flow should be:
1. Disable all OSDs
2. Update Version
3. Fix Ownership
4. Enable all OSDs
5. Restart all OSDs
The disable should be on the leading edge in case the update or ownership
changes get interrupted or fail. If we can't guarantee the OSDs are sourcing a
stable update with proper permissions, they shouldn't be allowed to start.