openstack-upgrade doesn't restart ceph-mgr@<hostname>.service after luminous to mimic

[Drew Freiberger]

I noticed after performing an openstack/ubuntu release upgrade from xenial-queens to bionic-stein (and the x->b, then bionic-rocky steps in between) that the output of ceph versions showed everything running mimic except for the ceph-mon units's "mgr" processes.

ubuntu@juju-c27f05-6-lxd-0:~$ sudo ceph versions
{
    "mon": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
    },
    "mgr": {
        "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 3
    },
    "osd": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 134
    },
    "mds": {},
    "rgw": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
    },
    "overall": {
        "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 3,
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 140
    }
}

restarting the ceph-mgr@$(hostname) service across the three mons one at a time solves this:

{
    "mon": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
    },
    "mgr": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
    },
    "osd": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 134
    },
    "mds": {},
    "rgw": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3
    },
    "overall": {
        "ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 143
    }
}

[Alex Kavanagh (ajkavanagh)]

The problem actually occurs when the upgrade is from distro->cloud:bionic-rocky (no need to proceed to stein to show the issue):

$ sudo ceph versions
{
    "mon": {
        "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)": 3
    },
    "mgr": {
        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 3
    },
    "osd": {
        "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)": 9
    },
    "mds": {},
    "overall": {
        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 3,
        "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)": 12
    }
}

versions:

$ ~/bin/show-origin-apps-in-model.py
app: ceph-mon origin: cloud:bionic-rocky
app: ceph-osd origin: cloud:bionic-rocky
app: cinder origin: distro
app: cinder-ceph origin: <not known>
app: glance origin: distro
app: keystone origin: distro
app: nova-cloud-controller origin: distro
app: nova-compute origin: distro
app: percona-cluster origin: distro
app: prometheus2 origin: <not known>
app: rabbitmq-server origin: distro

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/740010

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740010
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=5ea8db66c83810a397b093e130bef7b2492e9b14
Submitter: Zuul
Branch: master

commit 5ea8db66c83810a397b093e130bef7b2492e9b14
Author: Alex Kavanagh <email address hidden>
Date: Wed Jul 8 15:43:33 2020 +0100

    Fix upgrade bug (luminous) where mgr is not restarted

    From luminous to mimic, the ceph-mgr isn't restarted, leaving it at
    luminous, rather than mimic. This change ensures that the ceph-mgr is
    restarted when the version is luminous or later.

    Related charms.ceph change: I2844736af2de27ce071db859311ac99b9b3057ad
    Closes-Bug: #1849874

    Change-Id: Id059fbfd46bed4b9f96e85251fa83067696857d5