unhealthy juju-db is invisible

[Paul Collins]

I just discovered that the mongodb replicaset of one of our Juju controllers was unhealthy as I was preparing for an upgrade, because juju create-backup was attempted on a secondary (this is a bug in itself, LP:1790875, but it was helpful here) that was in RECOVERING state.

mongodb is a critical wad of dark matter underpinning Juju that we have to deal with directly in order to monitor and manage. Juju should at least have some way of making the operator aware of an unhealthy database so that they can start Googling feverishly for help on fixing it.

[Richard Harding (rharding)]

Thank you. This falls directly into the work we want to do around an easy to consume health report for Juju controllers, in particular around HA status and mongo status.

[Peter Jose De Sousa (pjds)]

hit this, the controller upgrade was failing stating one of the units was in state "RECOVERING".

To workaround this issue, check the logs of the units, finding the unit stating that Mongodb is too far behind to re-sync.

Then follow the steps on the mongodb docs to re-sync the DB.

https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/

[Canonical Juju QA Bot (juju-qa-bot)]

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.