jewel upgrade takes down pgs when chown takes >10 min

[Shane Peters]

Part of the upgrade process to jewel involves a recursive 'chown' on files under /var/lib/ceph. If this step takes longer than 10 minutes, the node is assumed dead and moves on upgrading the next node in the cluster.

Subsequently, if enough osd's are taken down for upgrade, it can bring down whole placement groups.
The following log shows this timeout being reached even though the chown is still being performed on the node.

juju log:
2018-04-09 15:56:27 INFO juju-log node-15 is not finished. Waiting
2018-04-09 15:56:27 INFO config-changed obtained 'osd_node-15_jewel_start'
2018-04-09 15:56:27 DEBUG worker.uniter.jujuc server.go:178 hook context id "ceph-osd/3-config-changed-2244033196891562383"; dir "/var/lib/juju/agents/unit-ceph-osd-3/charm"
2018-04-09 15:56:27 INFO juju-log Waited 10 mins on node node-15. current time: 1523288787.3321168 > previous node start time: 1523288768.8523903 Moving on

[James Page (james-page)]

I think general best practice would be to use the 'pause-health' and 'resume-health' actions on one on the ceph-mon units before commencing the ceph-osd upgrade; however there is another bug here in the fact that the charm moves on after 10 minutes of waiting - which is insufficient.

[James Page (james-page)]

OSD's are upgraded in turn so I think a general heartbeat timestamp from the current unit performing the upgrade is probably a good idea - for units with lots of osd and lots of data, 10 minutes is just to short...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/739535

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739535
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=6b0a11b4048027d7de11d28bbf388bd0c337400f
Submitter: Zuul
Branch: master

commit 6b0a11b4048027d7de11d28bbf388bd0c337400f
Author: Alex Kavanagh <email address hidden>
Date: Mon Jul 6 17:13:32 2020 +0100

    Add a progress watchdog for OSD upgrades

    This patch (in charms.ceph [1], copied here) add the concept of a
    watchdog to the upgrade_monitor so that the charm can achieve two
    objectives of 1. Waiting for much longer, but 2. detecting whether the
    previous node has died / gone away. This is needed for 'large' OSDs
    where the time to upgrade a node may exceed the current limit of 10
    minutes, but also not to wait for 30 minutes on a dead previous node.
    The watchdog implements two timeouts and an addition 'alive' key from
    the previous node to indicate that it is still running. Otherwise,
    functionality is identical.

    [1] See depends on below
    Depends-On: Ia450e936c2096f092af3be5a369b7abaf5023b16
    Closes-Bug: #1762852

    Change-Id: I6204a5ade684f0564c4be2d30df467c75baa6dba