controller upgrade fails to progress

[Tim Penhey]

An HA controller upgrade was requested. All three controllers downloaded the new agent binaries and restarted. All three controllers checked in to start the upgrade.

Machine 2 hit an error during the upgrade and failed. This left the upgrade in a broken state.

Problem was disk space, and after cleanup, agent was restarted.

Restarting the machine agent didn't fix the problem as the database upgrade worker failed to start.

2020-05-04 02:11:19 ERROR juju.worker.upgradedatabase worker.go:284 timed out waiting for primary database upgrade

Looking at the code, worker/upgradedatabase/worker.go, it seems that the worker is expecting a watcher change, and it didn't check the initial state of the upgrade info. Since the primary had already completed its upgrade, there was no watcher fired. Since the worker didn't get an update, it failed to progress, and aborted.

The fix should be to start the watcher, then look at the info. If the info is already in the right state, it should progress as if the change was signalled.

[Tim Penhey (thumper)]

https://github.com/juju/juju/pull/11530

[Drew Freiberger (afreiberger)]

This fix needs to be backported to 2.7.8 for those systems still running 2.6 and running upgrade-model to 2.7, or a workaround is needed to be documented in this bug.

[Pen Gale (pengale)]

We currently have no plans to release another 2.7 minor version, modulo major security patches.

We don't recommend upgrading to 2.7, however. 2.6 can be upgraded directly to 2.8, with no need to pass through intermediate versions.