Comment 3 for bug 1473517

I ran an experiment:

juju bootstrap a 1.18 system
juju deploy ubuntu -n 5
juju upgrade-juju --version 1.20.14

The upgrade worked and all agents on the deployed units upgraded to 1.20.14 and restarted.

Then I added a unit to the ubuntu service:

juju add-unit ubuntu

This resulted in a failure to provision a new machine:

  "6":
    agent-state-info: '(error: cannot make user data: invalid machine configuration:
      missing API hosts)'
    instance-id: pending
    series: trusty

For some reason, the 1.20.14 agents, on restart, failed to properly load and process a file called "provider-state" in the environment's control bucket. This meant that the jujud agent did not have the API host addresses available.

As a manual work around, I restarted the jujud agent on machine 0:

juju ssh 0
sudo service jujud-machine-0 restart

Then I could add unit.

To get the previous failed add units to work, I told juju to retry provisioning of those failed machines, eg

juju retry-provisioning 6

After a minute or so, Juju wakes up and retries the failed machines and the add unit operation from before completes.

This workaround fixed the observed issues on an upgraded 1.20.14 system coming from 1.18 in a simple example. It would need testing in a more complex environment to ensure it is robust. There are no plans for another 1.20 release.