CephOSD/Compute nodes crash under memory pressure unless custom tuned profile is used
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Medium
|
John Fulton |
Bug Description
Under a really heavy I/O load a hyperconverged OpenStack-Ceph cluster has collapsed, multiple times, almost immediately in a chain reaction:
- OOM killer runs
- random nova instances and ceph-osd processes are killed
- Ceph goes into recovery mode
- OSDs consume much more memory than before
- OSDs hit cgroup limit and crash
- remaining OSDs are under even more pressure than before, and so on.
- Ceph reaches a point where some data is inaccessible (not lost) due to too many OSDs down.
- VMs hang and eventually get stack traces stating that I/O timeouts of > 120 seconds are occurring.
The issue can be worked around by lowering the dirty ratio and disabling KSM. Here's a tuned profile:
[main]
summary=ceph-osd Filestore tuned profile
include=
[sysctl]
vm.dirty_ratio = 10
vm.dirty_
[sysfs]
/sys/kernel/
If this file is installed with OpenStack as /usr/lib/
# tuned-adm profile ceph-osd-hci
This will persistently make these changes to the kernel configuration.
The suggested tuned profile could be encoded in a map like this:
tuned_ custom_ profile:
value: ceph-osd Filestore tuned profile
value: throughput- performance
value: 10 background_ ratio
value: 3 mm/ksm/ run
value: 0
name: my_custom_profile
sections:
- name: main
params:
- option: summary
- option: include
- name: sysctl
params:
- option: vm.dirty_ratio
- option: vm.dirty_
- name: sysfs
params:
- option: /sys/kernel/