upgrade to luminous results in warning status

[Zachary Zehring]

[Impact]
After upgrading ceph-mon and ceph-osd from xenial-ocata to xenial-pike, ceph status gives HEALTH_WARN with a message reading "all OSDs are running luminous or later but require_osd_release < luminous". For an OK health status, require-osd-release needs to be at least luminous.

[Steps to Reproduce]
Deploy the following bundle: file located below.

Upgrade to ceph-mon and ceph-osd to pike:
- juju config ceph-mon source=cloud:xenial-pike
- juju config ceph-osd source=cloud:xenial-pike

Get ceph status:
- juju ssh ceph-mon/0 sudo ceph status -f json-pretty
(Alternatively)
- juju ssh ceph-mon/0 sudo /usr/local/lib/nagios/plugins/check_ceph_status.py

You will see the status is HEALTH_WARN and has the following note:
all OSDs are running luminous or later but require_osd_release < luminous

[Resolution]
To fix this issue, all that needs to be run is the following command:
- juju ssh ceph-mon/0 sudo ceph osd require-osd-release luminous
Source: https://ceph.com/community/new-luminous-upgrade-complete/

[Proposal]
So if this command can be run without regard for versions of ceph-osd, then you could potentially add it to the upgrade hook. If this isn't possible, an action seems appropriate.

[Chris MacNaughton (chris.macnaughton)]

This should certainly be able to be handled in the normal ceph upgrade code, but will take some thought on how to do it in the best way.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660210

[Zachary Zehring (zzehring)]

Also, here's a command for checking what require_osd_release is set to:

ceph osd dump | grep require_osd_release

The release output should match the release versions for all the osds.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660210
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=63b38bf5ce7080c0c1f7b815efaf41e3c862c309
Submitter: Zuul
Branch: master

commit 63b38bf5ce7080c0c1f7b815efaf41e3c862c309
Author: Zachary Zehring <email address hidden>
Date: Mon May 20 16:33:23 2019 -0400

    Add set require-osd-release command to osd hook.

     To access all ceph features for a new release,
     require-osd-release must be set to the current
     release. Else, features will not be available
     and ceph health gives a warning on luminous.
     Here, we check to see if an osd has upgraded its
     release and notified mon. If so, we run the
     post-upgrade steps when all osds have reached
     the new release. The one (and only) step
     is to set require-osd-release if and only if
     all osds (and mons) have been upgraded to the
     same version.

    Get osd release information from ceph_release key in
    relation dict.

    Add call to set require-osd-release to current release.

    Add execute post-upgrade steps func in osd-relations hook.

    Add logic for determinig whether to run set
    require-osd-release command.

    Add logic for checking if all osds and mons have
    converged to same release.

    Create func to grab all unique osd releases on each unit.

    Change-Id: Ia0bc15b3b6d7e8a21fda8e2343d70d9a0024a767
    Closes-Bug: #1828630