OSDs unable to connect to monitor after ceph upgrade

[Billy Olsen]

OSDs were unable to connect to monitors after an upgrade from nautilus to octopus. It turns out, the monitors had the require-osd-release set to mimic. It is documented in upstream documentation that the upgrade from mimic or nautilus [0], but this failed for some unknown reason.

This was resolved by running `ceph osd set-require-osd-release octopus`. However, the charms should have the ability to set the current require-osd-release setting for Ceph after an upgrade. Additionally, this should be documented in the upgrade process.

[0] - https://docs.ceph.com/en/latest/releases/octopus/#upgrading-from-mimic-or-nautilus

[Billy Olsen (billy-olsen)]

Note that the symptoms experienced are the same as identified in bug #1874939, however I am keeping it separate for the time being as that was primarily focused on packaging level issues and the upgrade. It is possible this bug will become a duplicate of that bug and it is fixed there instead, but for now I am keeping this focused on the charms.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-deployment-guide/+/792479
Committed: https://opendev.org/openstack/charm-deployment-guide/commit/628556205c610b86ebfc58f106e3df42379fc16b
Submitter: "Zuul (22348)"
Branch: master

commit 628556205c610b86ebfc58f106e3df42379fc16b
Author: Peter Matulis <email address hidden>
Date: Thu May 20 15:58:51 2021 -0400

    Add Ceph upgrade issue - require-osd-release

    Issue found during the ServerStack upgrade.

    The ceph-mon charm should be able to validate and
    respond intelligently to the 'require-osd-release'
    option's value.

    Partial-Bug: #1929254
    Change-Id: I2afca1812479d18880dcf7144ae3a456ce112c57

[Trent Lloyd (lathiat)]

I think this bug documents the same issue:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1874939

My comment there

"This issue appears to be documented here: https://docs.ceph.com/en/latest/releases/nautilus/#instructions

Complete the upgrade by disallowing pre-Nautilus OSDs and enabling all new Nautilus-only functionality:

# ceph osd require-osd-release nautilus
Important This step is mandatory. Failure to execute this step will make it impossible for OSDs to communicate after msgrv2 is enabled."