Agents fail to upgrade to 2.2.7, unable to 'juju run'

[Paul Gear]

After an apparently-successful upgrade of a controller from 2.2.6 to 2.2.7 (https://pastebin.canonical.com/205786/) and an upgrade request for all hosted models, all machine & unit agents in the hosted model fail to upgrade to 2.2.7, and 'juju run' is no longer functional, reporting (after around 5 minutes):

ERROR timed out waiting for results from: machine 0, machine 1, machine 2, machine 3, machine 4, machine 5, machine 6

[Paul Gear (paulgear)]

Diagnostics:

- juju-goroutines from controller machine agent: https://pastebin.canonical.com/205795/

- juju-engine-report from controller machine agent: https://pastebin.canonical.com/205796/

- Complete logs from controller: https://private-fileshare.canonical.com/~paulgear/lp1738728/

- juju-goroutines from hosted model machine agent: https://pastebin.canonical.com/205790/

- juju-engine-report from hosted model machine agent: https://pastebin.canonical.com/205791/

- Complete logs from hosted model machine agent: https://pastebin.canonical.com/205794/

[William Grant (wgrant)]

I experimented a little on my affected controller. Two rounds of full model agent restarts got things upgraded (the first round upgraded the machine agents, and then the second round upgraded the unit agents, despite all being restarted each time). But the agents still wouldn't react to anything, even after a further controller restart.

I investigated the odd behaviour further with "juju run". An action on any agent on the controller normally times out, but can be made to succeed if the target agent is restarted while the action is pending. It seems that the 2.2.7 controller is always able to deliver up to date state to its agents on their startup, but it can't deliver events.

[William Grant (wgrant)]

We downgraded my model and controller to 2.2.6. The model is easy, the controller is a bit harder:

For the model:

 - In a mongo shell on the controller:
   db.settings.update({"_id": "$MODEL_UUID:e"}, {$set: {"settings.agent-version": "2.2.6"}})

 - Restart all model machine agents.

 - Wait a few seconds and restart all model unit agents.

 - "juju status --format yaml | grep version" should now report 2.2.6 across the board.

For the controller:

 - In a mongo shell on the controller:
   db.settings.update({"_id": "$CONTROLLER_MODEL_UUID:e"}, {$set: {"settings.agent-version": "2.2.6"}})

 - In each controller machine's agent.conf, change upgradedToVersion to 2.2.6.

 - Restart all controller machine agents.

 - Restart any controller unit agents.

And you should be back in business.

It may also be desirable to purge the 2.2.7 upgradeInfo, but that wasn't necessary for the downgrade.

[Tim Penhey (thumper)]

https://github.com/juju/juju/pull/8237

[Tim Penhey (thumper)]

https://github.com/juju/juju/pull/8238

[Anastasia (anastasia-macmood)]

I am sure that the above PRs have been merged into develop (2.4+) as part of a bigger development merge. Hence, I am marking this report as Fix Committed.