# dpkg -l | grep -i ceph
ii ceph 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 distributed storage and file system
ii ceph-base 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 common ceph daemon libraries and management tools
ii ceph-common 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 manager for the ceph distributed file system
ii ceph-mon 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 monitor server for the ceph storage system
ii ceph-osd 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 OSD server for the ceph storage system
ii libcephfs2 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 Ceph distributed file system client library
ii python3-ceph-argparse 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 Python 3 utility libraries for Ceph CLI
ii python3-cephfs 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 Python 3 libraries for the Ceph libcephfs library
ii python3-rados 14.2.9-0ubuntu0.19.10.1~cloud0 amd64 Python 3 libraries for the Ceph librados library
# dpkg -l | grep -i ceph
ii ceph 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 distributed storage and file system
ii ceph-base 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 common ceph daemon libraries and management tools
ii ceph-common 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mds 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 metadata server for the ceph distributed file system
ii ceph-mgr 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 manager for the ceph distributed file system
ii ceph-mgr-modules-core 15.2.3-0ubuntu0.20.04.2~cloud0 all ceph manager modules which are always enabled
ii ceph-mon 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 monitor server for the ceph storage system
ii ceph-osd 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 OSD server for the ceph storage system
ii libcephfs2 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 Ceph distributed file system client library
ii python3-ceph-argparse 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 Python 3 utility libraries for Ceph CLI
ii python3-ceph-common 15.2.3-0ubuntu0.20.04.2~cloud0 all Python 3 utility libraries for Ceph
ii python3-cephfs 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 Python 3 libraries for the Ceph libcephfs library
ii python3-rados 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 Python 3 libraries for the Ceph librados library
ii python3-rbd 15.2.3-0ubuntu0.20.04.2~cloud0 amd64 Python 3 libraries for the Ceph librbd library
Then tested cluster as noted in the description (set `osd_inject_bad_map_crc_probability` to 1 on one OSD and then restart a different OSD). And observed no OSD crash happened and the cluster is OK.
I have tested this ussuri-proposed packages and it fixes the issue.
Setup a Nautilus cluster with the following versions:
# ceph versions ba46ee232b73b99 0f06029a2a0) nautilus (stable)": 1 ba46ee232b73b99 0f06029a2a0) nautilus (stable)": 1 ba46ee232b73b99 0f06029a2a0) nautilus (stable)": 3 ba46ee232b73b99 0f06029a2a0) nautilus (stable)": 5
{
"mon": {
"ceph version 14.2.9 (581f22da52345d
},
"mgr": {
"ceph version 14.2.9 (581f22da52345d
},
"osd": {
"ceph version 14.2.9 (581f22da52345d
},
"mds": {},
"overall": {
"ceph version 14.2.9 (581f22da52345d
}
}
# dpkg -l | grep -i ceph 0ubuntu0. 19.10.1~ cloud0 amd64 distributed storage and file system 0ubuntu0. 19.10.1~ cloud0 amd64 common ceph daemon libraries and management tools 0ubuntu0. 19.10.1~ cloud0 amd64 common utilities to mount and interact with a ceph storage cluster 0ubuntu0. 19.10.1~ cloud0 amd64 manager for the ceph distributed file system 0ubuntu0. 19.10.1~ cloud0 amd64 monitor server for the ceph storage system 0ubuntu0. 19.10.1~ cloud0 amd64 OSD server for the ceph storage system 0ubuntu0. 19.10.1~ cloud0 amd64 Ceph distributed file system client library ceph-argparse 14.2.9- 0ubuntu0. 19.10.1~ cloud0 amd64 Python 3 utility libraries for Ceph CLI 0ubuntu0. 19.10.1~ cloud0 amd64 Python 3 libraries for the Ceph libcephfs library 0ubuntu0. 19.10.1~ cloud0 amd64 Python 3 libraries for the Ceph librados library
ii ceph 14.2.9-
ii ceph-base 14.2.9-
ii ceph-common 14.2.9-
ii ceph-mgr 14.2.9-
ii ceph-mon 14.2.9-
ii ceph-osd 14.2.9-
ii libcephfs2 14.2.9-
ii python3-
ii python3-cephfs 14.2.9-
ii python3-rados 14.2.9-
Then upgraded the cluster to:
# ceph versions c1f516e0a093594 580a76b78d0) octopus (stable)": 1 c1f516e0a093594 580a76b78d0) octopus (stable)": 1 c1f516e0a093594 580a76b78d0) octopus (stable)": 3 c1f516e0a093594 580a76b78d0) octopus (stable)": 5
{
"mon": {
"ceph version 15.2.3 (d289bbdec69ed7
},
"mgr": {
"ceph version 15.2.3 (d289bbdec69ed7
},
"osd": {
"ceph version 15.2.3 (d289bbdec69ed7
},
"mds": {},
"overall": {
"ceph version 15.2.3 (d289bbdec69ed7
}
}
# dpkg -l | grep -i ceph 0ubuntu0. 20.04.2~ cloud0 amd64 distributed storage and file system 0ubuntu0. 20.04.2~ cloud0 amd64 common ceph daemon libraries and management tools 0ubuntu0. 20.04.2~ cloud0 amd64 common utilities to mount and interact with a ceph storage cluster 0ubuntu0. 20.04.2~ cloud0 amd64 metadata server for the ceph distributed file system 0ubuntu0. 20.04.2~ cloud0 amd64 manager for the ceph distributed file system modules- core 15.2.3- 0ubuntu0. 20.04.2~ cloud0 all ceph manager modules which are always enabled 0ubuntu0. 20.04.2~ cloud0 amd64 monitor server for the ceph storage system 0ubuntu0. 20.04.2~ cloud0 amd64 OSD server for the ceph storage system 0ubuntu0. 20.04.2~ cloud0 amd64 Ceph distributed file system client library ceph-argparse 15.2.3- 0ubuntu0. 20.04.2~ cloud0 amd64 Python 3 utility libraries for Ceph CLI 0ubuntu0. 20.04.2~ cloud0 all Python 3 utility libraries for Ceph 0ubuntu0. 20.04.2~ cloud0 amd64 Python 3 libraries for the Ceph libcephfs library 0ubuntu0. 20.04.2~ cloud0 amd64 Python 3 libraries for the Ceph librados library 0ubuntu0. 20.04.2~ cloud0 amd64 Python 3 libraries for the Ceph librbd library
ii ceph 15.2.3-
ii ceph-base 15.2.3-
ii ceph-common 15.2.3-
ii ceph-mds 15.2.3-
ii ceph-mgr 15.2.3-
ii ceph-mgr-
ii ceph-mon 15.2.3-
ii ceph-osd 15.2.3-
ii libcephfs2 15.2.3-
ii python3-
ii python3-ceph-common 15.2.3-
ii python3-cephfs 15.2.3-
ii python3-rados 15.2.3-
ii python3-rbd 15.2.3-
Then tested cluster as noted in the description (set `osd_inject_ bad_map_ crc_probability ` to 1 on one OSD and then restart a different OSD). And observed no OSD crash happened and the cluster is OK.