Upgrade from 1.21.3 -> 1.25.0 got minor errors

[Paolo de Rosa]

[Environment]

Juju core 1.21.3
Trusty 14.04.2

[Description]

The customer upgraded from 1.21.3 to 1.25.0 sequentially following the next upgrade plan:

{ "_id" : "1.25.0-trusty-amd64", "version" : "1.25.0-trusty-amd64", "size" : NumberLong(18672448), "sha256" : "0419fcde2d571a00eddd0b2fb9e35f5102a9fd81cbecd4a2d0b648da7740eab3", "path" : "tools/1.25.0-trusty-amd64-0419fcde2d571a00eddd0b2fb9e35f5102a9fd81cbecd4a2d0b648da7740eab3", "txn-revno" : NumberLong(2), "txn-queue" : [ "5658154471cf1104660055ff_2be272b4" ] }

The upgrade seems to be successful because it is able to operate almost all commands but It cannot get status of the units so charms upgrade fails.

juju@host1:~$ juju run --unit cinder/2 "status-get"
error: finding workload status: cannot get status: unit not found

juju@host1:~$ juju status cinder/2
ERROR could not filter units: could not filter units: cannot get status: unit not found

while juju status works

   cinder/2:
        workload-status:
          current: error
          message: 'hook failed: "config-changed"'
          since: 27 Nov 2015 19:25:26+01:00
        agent-status:
          current: idle
          since: 27 Nov 2015 19:25:26+01:00
          version: 1.25.0
        agent-state: error
        agent-state-info: 'hook failed: "config-changed"'
        agent-version: 1.25.0
        machine: 3/lxc/2
        public-address: 172.27.4.49
        subordinates:
          cinder-hacluster/2:
            workload-status: {}
            agent-status:
              current: idle
              since: 27 Nov 2015 19:23:11+01:00
              version: 1.25.0
            agent-state: started
            agent-version: 1.25.0
            upgrading-from: local:trusty/hacluster-69
            public-address: 172.27.4.49

the unit is in error state because upgrade-charm fail due to the same error
2015-11-27 17:36:55 INFO config-changed error: finding workload status: cannot get status: unit not found

attached machine-0 logs

[Mario Splivalo (mariosplivalo)]

I was able to reproduce this.
I downloaded juju 1.21.3 from https://launchpad.net/juju-core/+milestone/1.21.3, compiled it locally and used it to bootstrap my environment and then deploy some services:

http://paste.ubuntu.com/13530133/

Then I deployed some services:

ubuntu@mariosplivalo-bastion:~/dl/juju-core_1.21.3/bin$ ./juju deploy percona-cluster -n3
Added charm "cs:trusty/percona-cluster-31" to the environment.
ubuntu@mariosplivalo-bastion:~/dl/juju-core_1.21.3/bin$ ./juju deploy keystone -n2
Added charm "cs:trusty/keystone-31" to the environment.

After a while the services are installed:

ubuntu@mariosplivalo-bastion:~/dl/juju-core_1.21.3/bin$ ./juju status
environment: mariosplivalo
machines:
  "0":
    agent-state: started
    agent-version: 1.21.3
    dns-name: 10.5.3.38
    instance-id: e346bdb9-3f25-40c7-a9d1-b9ebbc2714ed
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.21.3
    dns-name: 10.5.3.39
    instance-id: ce4d47bc-314a-476f-b6b3-0c9bb805816e
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
  "2":
    agent-state: started
    agent-version: 1.21.3
    dns-name: 10.5.3.40
    instance-id: f6d6b09a-8d76-490f-822e-2b40233a5879
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
  "3":
    agent-state: started
    agent-version: 1.21.3
    dns-name: 10.5.3.41
    instance-id: f5b98fab-280e-4c55-a810-061069637618
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
  "4":
    agent-state: started
    agent-version: 1.21.3
    dns-name: 10.5.3.42
    instance-id: d8b1b8ef-1dcd-40ad-8140-8bb988a2aee0
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
  "5":
    agent-state: started
    agent-version: 1.21.3
    dns-name: 10.5.3.43
    instance-id: 94bb3854-0ae9-418a-974e-bfb7af561716
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
services:
  keystone:
    charm: cs:trusty/keystone-31
    exposed: false
    relations:
      cluster:
      - keystone
    units:
      keystone/0:
        agent-state: started
        agent-version: 1.21.3
        machine: "4"
        public-address: 10.5.3.42
      keystone/1:
        agent-state: started
        agent-version: 1.21.3
        machine: "5"
        public-address: 10.5.3.43
  percona-cluster:
    charm: cs:trusty/percona-cluster-31
    exposed: false
    relations:
      cluster:
      - percona-cluster
    units:
      percona-cluster/0:
        agent-state: started
        agent-version: 1.21.3
        machine: "1"
        public-address: 10.5.3.39
      percona-cluster/1:
        agent-state: started
        agent-version: 1.21.3
        machine: "2"
        public-address: 10.5.3.40
      percona-cluster/2:
        agent-state: started
        agent-version: 1.21.3
        machine: "3"
        public-address: 10.5.3.41

I then proce...

[Mario Splivalo (mariosplivalo)]

Attaching all-machines.log and machine-0.log from the bootstrap node.

[Mario Splivalo (mariosplivalo)]

And here's the machine-0.log