upgrade to luminous results in warning status

Bug #1828630 reported by Zachary Zehring
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph Monitor Charm
Fix Released
High
Zachary Zehring

Bug Description

[Impact]
After upgrading ceph-mon and ceph-osd from xenial-ocata to xenial-pike, ceph status gives HEALTH_WARN with a message reading "all OSDs are running luminous or later but require_osd_release < luminous". For an OK health status, require-osd-release needs to be at least luminous.

[Steps to Reproduce]
Deploy the following bundle: file located below.

Upgrade to ceph-mon and ceph-osd to pike:
- juju config ceph-mon source=cloud:xenial-pike
- juju config ceph-osd source=cloud:xenial-pike

Get ceph status:
- juju ssh ceph-mon/0 sudo ceph status -f json-pretty
(Alternatively)
- juju ssh ceph-mon/0 sudo /usr/local/lib/nagios/plugins/check_ceph_status.py

You will see the status is HEALTH_WARN and has the following note:
all OSDs are running luminous or later but require_osd_release < luminous

[Resolution]
To fix this issue, all that needs to be run is the following command:
- juju ssh ceph-mon/0 sudo ceph osd require-osd-release luminous
Source: https://ceph.com/community/new-luminous-upgrade-complete/

[Proposal]
So if this command can be run without regard for versions of ceph-osd, then you could potentially add it to the upgrade hook. If this isn't possible, an action seems appropriate.

description: updated
description: updated
Revision history for this message
Zachary Zehring (zzehring) wrote :
description: updated
Ryan Beisner (1chb1n)
Changed in charm-ceph-mon:
importance: Undecided → High
milestone: none → 19.07
summary: - ocata to pike upgrade results in warning status
+ upgrade to luminous results in warning status
Changed in charm-ceph-mon:
status: New → Confirmed
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

This should certainly be able to be handled in the normal ceph upgrade code, but will take some thought on how to do it in the best way.

Changed in charm-ceph-mon:
importance: High → Medium
Changed in charm-ceph-mon:
importance: Medium → High
Changed in charm-ceph-mon:
assignee: nobody → Zachary Zehring (zzehring)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-mon (master)

Fix proposed to branch: master
Review: https://review.opendev.org/660210

Revision history for this message
Zachary Zehring (zzehring) wrote :

Also, here's a command for checking what require_osd_release is set to:

ceph osd dump | grep require_osd_release

The release output should match the release versions for all the osds.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-mon (master)

Reviewed: https://review.opendev.org/660210
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=63b38bf5ce7080c0c1f7b815efaf41e3c862c309
Submitter: Zuul
Branch: master

commit 63b38bf5ce7080c0c1f7b815efaf41e3c862c309
Author: Zachary Zehring <email address hidden>
Date: Mon May 20 16:33:23 2019 -0400

    Add set require-osd-release command to osd hook.

     To access all ceph features for a new release,
     require-osd-release must be set to the current
     release. Else, features will not be available
     and ceph health gives a warning on luminous.
     Here, we check to see if an osd has upgraded its
     release and notified mon. If so, we run the
     post-upgrade steps when all osds have reached
     the new release. The one (and only) step
     is to set require-osd-release if and only if
     all osds (and mons) have been upgraded to the
     same version.

    Get osd release information from ceph_release key in
    relation dict.

    Add call to set require-osd-release to current release.

    Add execute post-upgrade steps func in osd-relations hook.

    Add logic for determinig whether to run set
    require-osd-release command.

    Add logic for checking if all osds and mons have
    converged to same release.

    Create func to grab all unique osd releases on each unit.

    Change-Id: Ia0bc15b3b6d7e8a21fda8e2343d70d9a0024a767
    Closes-Bug: #1828630

Changed in charm-ceph-mon:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-ceph-mon:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.