- OSD journals reside on rotating hard drives within the filesystem which stores the actual data [1],
which makes bandwidth at least 2x smaller (due to lots of fsync's) and latency 4x -- 10x bigger
due to numerious seeks,
- OSDs are co-hosted with hypervisors which is a bad idea (both need quite a lot of RAM and CPU)
In order to avoid performance problems please
- Put OSD journals on SSD, use a dedicated partition/logical volume instead of a file
- Allocate nodes specifically for OSDs, that is, don't co-host hypervisors/databases/etc there
ceph cluster is badly misconfigured:
- OSD journals reside on rotating hard drives within the filesystem which stores the actual data [1],
which makes bandwidth at least 2x smaller (due to lots of fsync's) and latency 4x -- 10x bigger
due to numerious seeks,
- OSDs are co-hosted with hypervisors which is a bad idea (both need quite a lot of RAM and CPU)
In order to avoid performance problems please
- Put OSD journals on SSD, use a dedicated partition/logical volume instead of a file databases/ etc there
- Allocate nodes specifically for OSDs, that is, don't co-host hypervisors/
[1] From node-51/ var/log/ ceph/ceph- osd.0.log
2016-04-25 06:49:10.141975 7f0e6532a800 1 journal _open /var/lib/ ceph/osd/ ceph-0/ journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 0 ceph/osd/ ceph-0/ journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 0
2016-04-25 06:49:10.142203 7f0e6532a800 1 journal _open /var/lib/