upgrade-juju: success but then deploy fails

[Roger Peppe]

I did this:

    $ cd $GOPATH/src/github.com/juju/juju
    $ go install ./...
    $ juju bootstrap --upload-tools --debug ec2 aws
    $ # oops, needed to make another (minor) change to the source
    $ go install ./...
    $ juju switch admin
    local.ec2:default -> local.ec2:admin
    $ juju upgrade-juju --upload-tools
    available tools:
        2.0-rc1.2-trusty-amd64
    best version:
        2.0-rc1.2
    $ # oops, yet another (minor) change required
    $ go install ./...
    $ juju upgrade-juju --upload-tools
    available tools:
        2.0-rc1.3-trusty-amd64
    best version:
        2.0-rc1.3
    $ juju switch default
    $ juju deploy somecharm

The instance for the new charm came up OK, but the machine agent
couldn't start because it failed to download the tools.

Some relevant lines from the machine agent's log on the newly started instance:

2016-04-15 14:28:34 DEBUG juju.worker.dependency engine.go:465 "upgrader" manifold worker started
2016-04-15 14:28:34 INFO juju.worker.upgrader upgrader.go:139 abort check blocked until version event received
2016-04-15 14:28:34 INFO juju.worker.upgrader upgrader.go:145 unblocking abort check
2016-04-15 14:28:34 INFO juju.worker.upgrader upgrader.go:178 desired tool version: 2.0-rc1.1
2016-04-15 14:28:34 INFO juju.worker.upgrader upgrader.go:199 upgrade requested from 2.0-rc1.3 to 2.0-rc1.1
2016-04-15 14:28:34 DEBUG juju.worker.dependency engine.go:465 "upgrade-steps-runner" manifold worker started
2016-04-15 14:28:34 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/juju/wrench: no such file or directory
2016-04-15 14:28:34 DEBUG juju.worker.dependency engine.go:479 "upgrade-steps-runner" manifold worker stopped: <nil>
2016-04-15 14:28:34 INFO juju.worker.upgrader upgrader.go:251 fetching tools from "https://10.169.42.56:17070/model/9c357d76-4db2-49fb-8b82-81e8b26aa08f/tools/2.0-rc1.1-trusty-amd64"
2016-04-15 14:28:36 ERROR juju.worker.upgrader upgrader.go:222 failed to fetch tools from "https://10.169.42.56:17070/model/9c357d76-4db2-49fb-8b82-81e8b26aa08f/tools/2.0-rc1.1-trusty-amd64": bad HTTP response: 400 Bad Request

The actual error returned from the above HTTP request was:

{"Error":{"Message":"error fetching tools: no matching tools available","Code":"bad request"}}

[Cheryl Jennings (cherylj)]

We shouldn't be getting rid of old tools if any hosted models are currently at that revision.

[Eric Snow (ericsnowcurrently)]

FYI, I ran into this too. The solution?

  juju upgrade-juju -m default 2.0-rc1.3

In other words, upgrading juju on your controller does *not* automatically upgrade juju on your models. The question is, should it? Not necessarily. I can imagine use cases where the admin might not want to automatically upgrade all models. Regardless here are the options I see:

1. do nothing (rely on users to know to upgrade juju per-model)
2. clearly point this out in the documentation
3. add a note in the output of "juju upgrade-juju --upload-tools" reminding admins to manually upgrade each model
4. make the "juju is out-of-date" warnings also show up for models relative to the controller
5. auto-upgrade models when the controller is upgraded
6. auto-upgrade but have a flag to not auto-upgrade
7. have a flag to auto-upgrade

[Eric Snow (ericsnowcurrently)]

@Roger, did you have any problem once the you upgraded the model too?

[Eric Snow (ericsnowcurrently)]

Also, I'll point out that this issue was very frustrating to me when I hit it and it was not obvious for quite a while what was going on. I expect it will bite devs more often than regular users, but it will still bite regular users. I don't think that doing nothing (#1) is an acceptable option. But I don't think that always auto-upgrading (#5) is acceptable either.

[Eric Snow (ericsnowcurrently)]

FWIW, I'll take this up with the juju-dev list tomorrow.

[Eric Snow (ericsnowcurrently)]

FTR, a related fix: https://github.com/juju/juju/pull/5226. That patch exposes all tools on the controller to the models. It landed 2 days ago, 5 days after this one was opened. However, it does not address the question of auto-upgrading models.

[Eric Snow (ericsnowcurrently)]

I realized there is a separate issue at play. Apparently the machine was provisioned at the (uploaded) version on the controller, rather than the model's.

This problem is exacerbated when the upgrader runs into the apparent downgrade to the model's version. The model did not have the original tools available, which it certainly should since it is that is the version it is running. However, this matter should be resolved by the previously referenced patch.

So at this point the reported trouble should no longer happen. I'm posting to juju-dev today about auto-upgrading models. I've opened lp:1573742 to address the matter of machines being initially provisioned to the admin model's version.

@Roger, please let us know if you're still having a problem with this.