Upgrade 1.24.7 -> 1.25.3 fails

[Stuart Bishop]

An attempt to upgrade an OpenStack environment with manually provisioned machines failed, with Machine 0 in an error state and all other units 'lost'.

upgrade to 1.25.3 failed (giving up): tag machine instances: tagging instance manual:xxxxxx.internal for machine 4: setting server metadata: failed to set metadata map[juju-env-uuid:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] on server with id: manual:xxxxxx.internal
      caused by: Resource at http://10.99.99.99:8774/v2/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/servers/manual:xxxxxx.internal/metadata not found
      caused by: request (http://10.99.99.99:8774/v2/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/servers/manual:xxxxxx.internal/metadata) returned unexpected status: 404; error info: {"itemNotFound": {"message": "Server does not exist", "code": 404}}

I will need guidance on how to recover the environment.

[Cheryl Jennings (cherylj)]

Relevant comment from IRC: The IP in the URLs is the address of the controller unit, and that is just openstack. I suspect the trigger is my units are all manually provisioned.

[Andrew Wilkins (axwalk)]

Yes, the issue is that they're manually provisioned. There's an oversight in upgrades/tags.go: we're passing all instances to the provider to add tags, but we should exclude any manual machines.

[Anastasia (anastasia-macmood)]

Whilst this bug is fixed, the ability to restart a broken upgrade needs to be addressed - bug # 1548564.

[Anastasia (anastasia-macmood)]

We are working on the upgrade path to recover the system in the original bug report.

[Anastasia (anastasia-macmood)]

At this stage, recovery is of environments with failed upgrade is manual. I'll describe the process in the bug that tracks recovery (bug # 1548564).

[Cheryl Jennings (cherylj)]

Users should upgrade directly to 1.25.4 to avoid hitting this issue.