juju upgrade from 1.24-beta2 to 1.24.5 broken

[Jacek Nykis]

I tried upgrading juju from 1.24-beta2 to 1.24.5 but it got stuck. First it was due to known problem:

https://lists.ubuntu.com/archives/juju/2015-June/005441.html

I worked around it by recreating symlink as per instructions above. This allowed upgrade to get a bit further but eventually upgrade on the bootstrap node but into a loop, showing this every few minutes:

2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: add block device documents for existing machines
2015-09-08 15:51:22 INFO juju.state open.go:51 opening state, mongo addresses: ["127.0.0.1:37017"]; entity machine-0
2015-09-08 15:51:22 DEBUG juju.state open.go:52 dialing mongo
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "172.31.23.113:37017"
2015-09-08 15:51:22 DEBUG juju.state open.go:57 connection established
2015-09-08 15:51:22 DEBUG juju.state open.go:64 mongodb login successful
2015-09-08 15:51:22 INFO juju.state state.go:182 creating lease client as machine-0
2015-09-08 15:51:22 INFO juju.state state.go:194 starting leadership manager
2015-09-08 15:51:22 INFO juju.state state.go:204 starting presence watcher
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:22 DEBUG juju.state open.go:293 closed state without error
2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: add instance id field to IP addresses
2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: migrate charm archives into environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:34 migrating charms to environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/haproxy-28, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/nrpe-external-master-7, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ntp-14, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ubuntu-repository-cache-1, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ubuntu-basenode-1, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ubuntu-repository-cache-2, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ksplice-3, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/landscape-client-5, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/haproxy-29, already in environment storage
2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: change entityid field on status history to globalkey
2015-09-08 15:51:22 INFO juju.state open.go:51 opening state, mongo addresses: ["127.0.0.1:37017"]; entity machine-0
2015-09-08 15:51:22 DEBUG juju.state open.go:52 dialing mongo
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:22 DEBUG juju.state open.go:57 connection established
2015-09-08 15:51:22 DEBUG juju.state open.go:64 mongodb login successful
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "172.31.23.113:37017"
2015-09-08 15:51:22 INFO juju.state state.go:182 creating lease client as machine-0
2015-09-08 15:51:22 INFO juju.state state.go:194 starting leadership manager
2015-09-08 15:51:22 INFO juju.state state.go:204 starting presence watcher
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:23 DEBUG juju.state open.go:293 closed state without error
2015-09-08 15:51:23 ERROR juju.upgrade upgrade.go:135 upgrade step "change entityid field on status history to globalkey" failed: environment UUID "xxxx": unexpected id: 0
2015-09-08 15:51:23 ERROR juju.cmd.jujud upgrade.go:359 upgrade from 1.24-beta2.1 to 1.24.5 for "machine-0" failed (will retry): change entityid field on status history to globalkey: environment UUID "xxxx": unexpected id: 0

juju status is now showing

  "0":
    agent-state: error
    agent-state-info: 'upgrade to 1.24.5 failed (giving up): change entityid field
      on status history to globalkey: environment UUID "xxxx":
      unexpected id: 0'
    agent-version: 1.24.5

[Martin Packman (gz)]

Looks like a regression from the fix for bug 1479289.

[Martin Packman (gz)]

Manual workaround got us past this:

Get a mongo shell per:

<https://gist.github.com/niedbalski/1306c7810e7c3aee9476>

Then delete all status history entries:

> db.statuseshistory.remove({})