Upgrade not possible from 1.25.6 to 1.25.10

[Jorge Niedbalski]

[Environment]

Trusty 14.04.2
Juju-core 1.25.6
Architecture: amd64.

[Description]

Is not possible to upgrade 1.25.6 to any further version.

[Steps to reproduce]

Initial deployment:

ubuntu@niedbalski-bastion:~$ juju status
environment: niedbalski
machines:
  "0":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: 10.5.0.50
    instance-id: dc07cd47-4a4a-4495-9270-04c2e754360b
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
    state-server-member-status: has-vote
services: {}

ubuntu@niedbalski-bastion:~$ juju set-env logging-config="<root>=TRACE"
ubuntu@niedbalski-bastion:~$ juju sync-tools

ubuntu@niedbalski-bastion:~$ juju --debug --verbose upgrade-juju --version="1.25.10"
2017-03-22 19:42:03 INFO juju.cmd supercommand.go:37 running juju [1.25.6-trusty-amd64 gc]
2017-03-22 19:42:03 DEBUG juju.api api.go:154 trying cached API connection settings - endpoints [10.5.0.50:17070]
2017-03-22 19:42:03 INFO juju.api api.go:266 connecting to API addresses: [10.5.0.50:17070]
2017-03-22 19:42:03 INFO juju.api apiclient.go:262 dialing "wss://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/api"
2017-03-22 19:42:03 INFO juju.api apiclient.go:194 connection established to "wss://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/api"
2017-03-22 19:42:03 DEBUG juju.api api.go:476 API hostnames unchanged - not resolving
2017-03-22 19:42:03 DEBUG juju.api api.go:506 cacheChangedAPIInfo: serverUUID="0a6ea588-7c12-4953-83bb-c0ca23979ea0"

2017-03-22 19:42:44 DEBUG juju.cmd.juju upgradejuju.go:187 available tools:
    1.25.10-centos7-amd64
    1.25.10-precise-amd64
    1.25.10-precise-arm64
    1.25.10-precise-ppc64el
    1.25.10-precise-s390x
    1.25.10-trusty-amd64
    1.25.10-trusty-arm64
    1.25.10-trusty-ppc64el
    1.25.10-trusty-s390x
    1.25.10-win10-amd64
    1.25.10-win2012-amd64
    1.25.10-win2012hv-amd64
    1.25.10-win2012hvr2-amd64
    1.25.10-win2012r2-amd64
    1.25.10-win7-amd64
    1.25.10-win8-amd64
    1.25.10-win81-amd64
    1.25.10-xenial-amd64
    1.25.10-xenial-arm64
    1.25.10-xenial-ppc64el
    1.25.10-xenial-s390x
    1.25.10-yakkety-amd64
    1.25.10-yakkety-arm64
    1.25.10-yakkety-ppc64el
    1.25.10-yakkety-s390x
2017-03-22 19:42:44 ERROR juju.cmd supercommand.go:429 connection is shut down

In the machine-0.log

2017-03-22 19:46:03 TRACE juju.apiserver apiserver.go:274 -> [1F] user-admin@local {"RequestId":4,"Response":{"List":[{"version":"1.15.0-precise-amd64","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.15.0-precise-amd64","sha256":"ea24934613691245dea26ba24df3e30fb5ba359dea2b03cc76c7aa0823dbfb23","size":3291685},{"version":"1.15.0-precise-i386","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.15.0-precise-i386","sha256":"b8c93942a154d9366a3a71d45534ae5a4276dd7e80cfbbd065387f87a659260d","size":3188608},{"version":"1.15.0-quantal-amd64","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.15.0-

[...]

,{"version":"1.25.9-xenial-ppc64el","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.25.9-xenial-ppc64el","sha256":"91a1f8b6f4ad1fc12fe1615c272c71bb3d12e1212a71f5e2e7466e7231997589","size":15317275},{"version":"1.25.9-yakkety-amd64","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.25.9-yakkety-amd64","sha256":"e34a226c1c85c548b299fe62dabe2e897296a0beebbc033698561c1dd977b7e7","size":16670573},{"version":"1.25.9-yakkety-arm64","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.25.9-yakkety-arm64","sha256":"5789cd287f48350ad09a1c668f700c7983d37c0c815365b6c8d586b7bf30d4c2","size":15532729},{"version":"1.25.9-yakkety-i386","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.25.9-yakkety-i386","sha256":"eff95b423ff4d1294bb766384beeccef2c24042b1299bf44c2dc3038b5b069da","size":15692980},{"version":"1.25.9-yakkety-ppc64el","url":"https://10.5.0.50:17070/environment/0a6ea588-7c12-4953-83bb-c0ca23979ea0/tools/1.25.9-yakkety-ppc64el","sha256":"0f4bd7438d83114021b1975fbfd3682d30e31d1ef95d63c2fcf1debce3da8998","size":15322051}],"Error":null}}

The following error is printed, and the upgrade didn't apply:

2017-03-22 19:46:03 TRACE juju.rpc.jsoncodec codec.go:112 <- error: EOF (closing false)

2017-03-22 19:46:03 TRACE juju.apiserver.common resource.go:117 stopping resource: logDir
2017-03-22 19:46:03 TRACE juju.apiserver.common resource.go:117 stopping resource: dataDir
2017-03-22 19:46:03 TRACE juju.apiserver.common resource.go:117 stopping resource: machineID
2017-03-22 19:46:03 INFO juju.apiserver apiserver.go:285 [1F] user-admin@local API connection terminated after 40.670557232s

[Felipe Reyes (freyes)]

I could reproduce this in another environment (openstack provider).

I grabbed the json printed in the reply before the EOF and it is a valid json no missing chars or anything. Here it is the reply pretty printed http://pastebin.ubuntu.com/24230526/

[Jorge Niedbalski (niedbalski)]

I followed down the issue LP: #1517632 , and comment https://bugs.launchpad.net/juju/+bug/1517632/comments/15 explains why the connection is being shutdown.

"This problem occurs when the distro-info on the client running the upgrade-juju command is backlevel compared with that of the state server. The state server will send back a list of tools that includes series that the client doesn't know about.

This will cause the unmarshal code to fail in the rpc layer, which causes the connection to shutdown. At this point, the client cannot request the upgrade as the connection has died."

Marking this issue as invalid, because I can upgrade the environment after upgrading the package distro-info and distro-info-data on the client side.

[Tim Penhey (thumper)]

No idea why this is happening at this stage. But can I get you to try to upgrade using a 1.25.10 client?

[John A Meinel (jameinel)]

As mentioned in the bug the issue is the client machine needs to know about
all series that the server might know about. While we may not spend the
time to fix it in 1.25 (the workaround is to upgrade your distro-info on
the client) we should make sure in the 2.x series we don't have the same
failure mode.

John
=:->

On Mar 23, 2017 03:10, "Tim Penhey" <email address hidden> wrote:

> No idea why this is happening at this stage. But can I get you to try to
> upgrade using a 1.25.10 client?
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1675154
>
> Title:
> Upgrade not possible from 1.25.6 to 1.25.10
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1675154/+subscriptions
>