Upgrade from 1.21.3 -> 1.22.8 -> 1.23.3 fails with 'ERROR a hosted environment cannot have a higher version than the server environment: 1.23.3.5 > 1.22.8.1'

[Jorge Niedbalski]

[Environment]

Trusty 14.04.2
Juju-core 1.21.3 (initially)

[Description]

I was trying to perform an upgrade to an environment from 1.21.3 -> 1.22.8 and then when I tried to upgrade
to 1.23.3 the error:

 'ERROR a hosted environment cannot have a higher version than the server environment: 1.23.3.5 > 1.22.8.1' is raised.

$ juju status
```yaml
environment: demo
machines:
  "0":
    agent-state: started
    agent-version: 1.21.3.1
    dns-name: 10.5.1.117
    instance-id: 96e22f1f-7ec7-4a06-98c3-bc70ddf63d7d
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.21.3.1
    dns-name: 10.5.1.118
    instance-id: 260356b9-6fbe-4bca-aad6-e398856f7350
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
  "2":
    agent-state: started
    agent-version: 1.21.3.1
    dns-name: 10.5.1.119
    instance-id: 9bd97a2f-979d-402d-9442-6b8fb226d2bc
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
  "3":
    agent-state: started
    agent-version: 1.21.3.1
    dns-name: 10.5.1.120
    instance-id: 0c47fdb2-e3ce-46b0-a990-ea5eaf89b1f3
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
services:
  percona-cluster:
    charm: local:trusty/percona-cluster-45
    exposed: false
    relations:
      cluster:
      - percona-cluster
    units:
      percona-cluster/0:
        agent-state: started
        agent-version: 1.21.3.1
        machine: "1"
        public-address: 10.5.1.118
      percona-cluster/1:
        agent-state: started
        agent-version: 1.21.3.1
        machine: "2"
        public-address: 10.5.1.119
      percona-cluster/2:
        agent-state: started
        agent-version: 1.21.3.1
        machine: "3"
        public-address: 10.5.1.120
```
$ juju upgrade-juju --version=1.22.8 --yes
```yaml
environment: demo
machines:
  "0":
    agent-state: started
    agent-version: 1.22.8.1
    dns-name: 10.5.1.127
    instance-id: 70c0de21-251d-47c3-9dae-a3b35dde4fa9
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.22.8.1
    dns-name: 10.5.1.128
    instance-id: e6ffa985-58fe-42b5-8e54-609291f6e413
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
  "2":
    agent-state: started
    agent-version: 1.22.8.1
    dns-name: 10.5.1.129
    instance-id: a02550ec-cf6a-4d11-9f7f-82534ddbf836
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
  "3":
    agent-state: started
    agent-version: 1.22.8.1
    dns-name: 10.5.1.130
    instance-id: f6aec45c-2f61-44ff-bf7b-c872ee51f601
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
services:
  percona-cluster:
    charm: local:trusty/percona-cluster-45
    exposed: false
    relations:
      cluster:
      - percona-cluster
    units:
      percona-cluster/0:
        agent-state: started
        agent-version: 1.22.8.1
        machine: "1"
        public-address: 10.5.1.128
      percona-cluster/1:
        agent-state: started
        agent-version: 1.22.8.1
        machine: "2"
        public-address: 10.5.1.129
      percona-cluster/2:
        agent-state: started
        agent-version: 1.22.8.1
        machine: "3"
        public-address: 10.5.1.130
```
$ juju upgrade-juju --version="1.23.3" --yes
```
2015-11-26 15:39:47 DEBUG juju.environs.tools build.go:232 forcing version to 1.23.3.1
2015-11-26 15:39:47 DEBUG juju.environs.tools build.go:38 adding entry: &tar.Header{Name:"FORCE-VERSION", Mode:436, Uid:0, Gid:0, Size:8, ModTime:time.Time{sec:63584149187, nsec:0x2b64c3c0, loc:(*time.Location)(0x44cfdc0)}, Typeflag:0x30, Linkname:"", Uname:"ubuntu", Gname:"ubuntu", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:63584149187, nsec:0x2b64c3c0, loc:(*time.Location)(0x44cfdc0)}, ChangeTime:time.Time{sec:63584149187, nsec:0x2b64c3c0, loc:(*time.Location)(0x44cfdc0)}, Xattrs:map[string]string(nil)}
2015-11-26 15:39:47 DEBUG juju.environs.tools build.go:38 adding entry: &tar.Header{Name:"jujud", Mode:493, Uid:0, Gid:0, Size:118017512, ModTime:time.Time{sec:63584149187, nsec:0x2b27bac0, loc:(*time.Location)(0x44cfdc0)}, Typeflag:0x30, Linkname:"", Uname:"ubuntu", Gname:"ubuntu", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:63584149187, nsec:0x2b27bac0, loc:(*time.Location)(0x44cfdc0)}, ChangeTime:time.Time{sec:63584149187, nsec:0x2b27bac0, loc:(*time.Location)(0x44cfdc0)}, Xattrs:map[string]string(nil)}
2015-11-26 15:39:58 INFO juju.environs.sync sync.go:319 built tools 1.23.3.1-trusty-amd64 (18158kB)
2015-11-26 15:39:58 INFO juju.cmd.juju upgradejuju.go:326 uploading tools 1.23.3.1-trusty-amd64 (18158kB) to Juju state server
```
$ juju upgrade-juju --version="1.23.3" --upload-tools

```bash
available tools:
    1.23.3.5-trusty-amd64
best version:
    1.23.3.5
ERROR a hosted environment cannot have a higher version than the server environment: 1.23.3.5 > 1.22.8.1
```

[Martin Packman (gz)]

This was an attempt to reproduce bug 1520314 that failed in a different way. Note that we do not suggest stepped upgrades through 1.23 as that has a number of known upgrade issues, but this doesn't seem to be related to the specific one that caused the earlier problems:

<https://lists.ubuntu.com/archives/juju/2015-June/005441.html>

[Cheryl Jennings (cherylj)]

I also ran into this while testing an environment which had been upgraded from 1.22.8 -> 1.25.0. I hit this error trying to upgrade after that.

[Cheryl Jennings (cherylj)]

There are two causes behind this issue (in my case from 1.22.8 -> 1.25.0, then attempting to upgrade again)

1 - The agent config file is updated after it is read from to connect to the state server. Specifically, there is an upgrade step to add the environment tag to the agent.conf file which happens after we already read from the file to connect and perform the upgrade. This environment tag is passed in when we open state and is recorded as the server tag. Since it is empty, later comparisons when attempting a follow on upgrade fail, leading us to believe we're operating on a hosted environment.

2 - The first issue should be okay, since we should handle the case of older agents, except that when opening state with an empty environment tag, the look up of the environment uuid from the database is used to set the environTag in state, but should also be used to set the serverTag.

The fix in this case is to use the uuid discovered from the database as the serverTag in cases where the environTag is empty.

You can work around this issue by restarting the machine agents. This will cause them to read in the environment tag from the agent.conf which was written after they restarted.

However, can you confirm the levels you're on where you're seeing this error? The code that has this check about hosted environments doesn't exist in 1.22 and 1.23.

[Jorge Niedbalski (niedbalski)]

This is not longer affecting any of our customers. So, flagging this bug as invalid.