Upgrade from 2.1.x to 2.2.1 blocked by missing Azure resources

[Paul Gear]

Upgrading a juju 2.1.x controller to 2.2.1 blocked several times with errors like the following:

2017-06-26 05:13:23 ERROR juju.worker.upgradesteps worker.go:375 upgrade from 2.1.3 to 2.2.1 for "machine-0" failed (will retry): Create common resource deployment: listing virtual machines: compute.VirtualMachinesClient#List: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'juju-mymodel-model-UUID' could not be found."

The only solution found was to remove these models manually from the controller using the following script: http://pastebin.ubuntu.com/24954497/

This script was invoked as "id=YOUR_UUID_HERE ./script" for each azure resource group which could not be found.

[Roger Peppe (rogpeppe)]

Is it possible that all these errors are because users have manually removed the model's resources rather than removing the model within juju itself?

[Paul Gear (paulgear)]

@rogpeppe: Totally possible; but we shouldn't be stopped from upgrading JAAS controllers by misbehaving users.

[Richard Harding (rharding)]

To be clear, this is in upgrading the controller and not the model itself?

[John A Meinel (jameinel)]

Correct. We changed how we tag some of the resources, and so we added an
Upgrade step to update the information in the cloud. However, if we no
longer have access to a particular user's information we can't update it,
and there is a bit of concern around how we want to handle that case. Also
some discussion around whether updated controllers can/should update cloud
information. (Do we want to have multiple possible DB/Cloud representations
to maintain compatibility, which adds a lot of internal complexity. The
Controller is still the place where the Provisioner lives, even if the
Agents have not been upgraded, the Provisioner has, and having multiple
code paths there would be hard.)

John
=:->

On Tue, Jun 27, 2017 at 4:07 AM, Richard Harding <<email address hidden>
> wrote:

> To be clear, this is in upgrading the controller and not the model
> itself?
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1700451
>
> Title:
> Upgrade from 2.1.x to 2.2.1 blocked by missing Azure resources
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1700451/+subscriptions
>

[Andrew Wilkins (axwalk)]

https://github.com/juju/juju/pull/7599
https://github.com/juju/juju/pull/7602

[Andrew Wilkins (axwalk)]

Last one, some observability: https://github.com/juju/juju/pull/7609