I noticed after performing an openstack/ubuntu release upgrade from xenial-queens to bionic-stein (and the x->b, then bionic-rocky steps in between) that the output of ceph versions showed everything running mimic except for the ceph-mon units's "mgr" processes.
ubuntu@juju-c27f05-6-lxd-0:~$ sudo ceph versions
{
"mon": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
},
"mgr": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 3
},
"osd": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 134
},
"mds": {},
"rgw": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
},
"overall": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 3,
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 140
}
}
restarting the ceph-mgr@$(hostname) service across the three mons one at a time solves this:
{
"mon": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
},
"mgr": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
},
"osd": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 134
},
"mds": {},
"rgw": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
},
"overall": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 143
}
}
The problem actually occurs when the upgrade is from distro- >cloud: bionic- rocky (no need to proceed to stein to show the issue):
$ sudo ceph versions 9cc913a0f5d3629 53a5d9c3ae0) mimic (stable)": 3 57dc0567da126be 145106aa47e) luminous (stable)": 3 9cc913a0f5d3629 53a5d9c3ae0) mimic (stable)": 9 57dc0567da126be 145106aa47e) luminous (stable)": 3, 9cc913a0f5d3629 53a5d9c3ae0) mimic (stable)": 12
{
"mon": {
"ceph version 13.2.8 (5579a94fafbc1f
},
"mgr": {
"ceph version 12.2.13 (584a20eb0237c6
},
"osd": {
"ceph version 13.2.8 (5579a94fafbc1f
},
"mds": {},
"overall": {
"ceph version 12.2.13 (584a20eb0237c6
"ceph version 13.2.8 (5579a94fafbc1f
}
}
versions:
$ ~/bin/show- origin- apps-in- model.py controller origin: distro
app: ceph-mon origin: cloud:bionic-rocky
app: ceph-osd origin: cloud:bionic-rocky
app: cinder origin: distro
app: cinder-ceph origin: <not known>
app: glance origin: distro
app: keystone origin: distro
app: nova-cloud-
app: nova-compute origin: distro
app: percona-cluster origin: distro
app: prometheus2 origin: <not known>
app: rabbitmq-server origin: distro