juju-upgrade from 2.1.2 to 2.3.2 fails

[Sandor Zeestraten]

# Issue
In the pursuit of a workaround for lp#1746265 and the recommendation of Rick in #juju-dev, I tried upgrading 3x HA juju controllers from 2.1.2 directly to 2.3.2.

As with lp#1746265, it seems that the juju environment becomes unusable and stuck in an upgrading state though from seemingly other reasons if I'm interpreting the controller machine logs correctly.

I managed to reproduce this every time both on MAAS and LXD.

# How to reproduce on LXD
* juju bootstrap lxd # with juju 2.1.2 client
* juju enable-ha
* juju upgrade-juju -m controller --agent-version 2.3.2 # with juju 2.3.2 client

Here's a repository with a Vagrant machine with steps to reproduce for LXD.

# Logs
See attached machine logs from a MAAS reproduction.

[Sandor Zeestraten (szeestraten)]

Forgot link to repository: https://github.com/szeestraten-ntnu/juju-2.1.2-to-2.3.2-upgrade

[Tim Penhey (thumper)]

Well... crap. Thank you for the logs, they were instrumental in showing exactly what was wrong.

Effectively something was added in 2.3.2 that relied on a model change in 2.3.0, but when upgrading from an earlier version, like 2.1 or 2.2, this hadn't been applied before it was expected.

[Tim Penhey (thumper)]

Not as crazy as I thought, as upgrading through 2.2 means that we don't hit this.

[Tim Penhey (thumper)]

https://github.com/juju/juju/pull/8356