Ability to abort/rollback broken controller upgrades
Bug #1761288 reported by
Sandor Zeestraten
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Triaged
|
Low
|
Unassigned |
Bug Description
Users needs to be able abort/rollback controller upgrades that fail or break.
This is a result of hitting multiple different scenarios where entire Juju environments were rendered unusable/
See #1746265 and #1748294 for examples scenarios.
Changed in juju: | |
status: | New → Triaged |
importance: | Undecided → Wishlist |
To post a comment you must log in.
I agree this would be nice to have. However, we do have "juju-force- upgrade" which can help depending on the cause of the upgrade failure. fixup/cleanup step required. And if we knew what that was going to be, then we would have fixed the original bug and not needed to rollback the upgrade.
However, most specific upgrade failures ultimately are causing a change to the database that is preventing us from moving forward, which means that there is *some* sort of manual intervention/
If this was deemed particularly critical, we could potentially do something like stop Mongo, copy the db to a backup location, copy the tools to an alternate location, and then have a rollback that would wipe out the active directory and move everything back to what it was.
All of this could be done outside of Juju itself.
The other option that we currently support is to bring up a different juju controller at the new version, and migrate models over to it one-by-one, which has more support for testing that it will be ok in the new location and rolling back if it isn't. However, it does require a temporary set of extra machines in order to do so.
We could support migrate-in-place by bootstrapping a new set of controllers on a different set of ports, with a different mongo cluster at different ports, and rotate them in. However, that would heavily complicate most sites that lock down ports that are accessible.