Comment 2 for bug 1664435

I've just stumbled upon this bug, and have to add a slightly connected issue with this:

if the upgrade is started with at least 1 ceph-osd node down, it will never add it's start_time to the KV, but the next node calculates its position and will wait until that DOWN node would come back. This effectively blocks the upgrade until we have that node repaired or removed from the cluster.

My suggestion is to check if the previous node is IN, and only consider those during the upgrade.
In case the DOWN node comes back during that time, it would anyway cause rebalancing, so I consider that the upgrade of that node would not cause much troubles during the upgrade (but it's rebalancing might). Also putting back nodes is rather an OPS task, and the engineer can decide whether it's acceptable to bring the node back during an upgrade or not. But blocking the upgrade indefinitely is not acceptable.

The code path that are effected (in def wait_on_previous_node(upgrade_key, service, previous_node, version):
        previous_node_start_time = monitor_key_get(
            upgrade_key,

Here the monitor_key_get errors out with this message in the logs:
2019-07-30 09:58:13 DEBUG config-changed Error ENOENT: error obtaining 'osd_wa0b1b-sto200j313m50-pl_jewel_start': (2) No such file or directory
2019-07-30 09:58:13 INFO juju-log Monitor config-key get failed with message: b''