Ability to abort/rollback broken controller upgrades

[Sandor Zeestraten]

Users needs to be able abort/rollback controller upgrades that fail or break.

This is a result of hitting multiple different scenarios where entire Juju environments were rendered unusable/unrecoverable after running `juju upgrade-juju -m controller`

See #1746265 and #1748294 for examples scenarios.

[John A Meinel (jameinel)]

I agree this would be nice to have. However, we do have "juju-force-upgrade" which can help depending on the cause of the upgrade failure.
However, most specific upgrade failures ultimately are causing a change to the database that is preventing us from moving forward, which means that there is *some* sort of manual intervention/fixup/cleanup step required. And if we knew what that was going to be, then we would have fixed the original bug and not needed to rollback the upgrade.
If this was deemed particularly critical, we could potentially do something like stop Mongo, copy the db to a backup location, copy the tools to an alternate location, and then have a rollback that would wipe out the active directory and move everything back to what it was.

All of this could be done outside of Juju itself.

The other option that we currently support is to bring up a different juju controller at the new version, and migrate models over to it one-by-one, which has more support for testing that it will be ok in the new location and rolling back if it isn't. However, it does require a temporary set of extra machines in order to do so.

We could support migrate-in-place by bootstrapping a new set of controllers on a different set of ports, with a different mongo cluster at different ports, and rotate them in. However, that would heavily complicate most sites that lock down ports that are accessible.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.