I had removed a node from the stack and the node was still listed in the CRUSH map. So, when the charms started upgrading one by one, I noticed one of the charms was waiting on another (that no longer existed). In CRUSH that bucket (host) and associated OSDs we're labeled down. I removed them, but there is currently no way (that I can see) to get the upgrades completed normally. I tried restarting the waiting nodes service (juju-unit-ceph-osd-*) and the waiting node continued, but it breaks the upgrade logic and the NEXT node in line doesnt get the notification that the ORIGINAL waiting node was completed.
This is still happening.
I had removed a node from the stack and the node was still listed in the CRUSH map. So, when the charms started upgrading one by one, I noticed one of the charms was waiting on another (that no longer existed). In CRUSH that bucket (host) and associated OSDs we're labeled down. I removed them, but there is currently no way (that I can see) to get the upgrades completed normally. I tried restarting the waiting nodes service (juju-unit- ceph-osd- *) and the waiting node continued, but it breaks the upgrade logic and the NEXT node in line doesnt get the notification that the ORIGINAL waiting node was completed.