cannot perform multiple upgrades

[Ian Booth]

Deploy a Juju 1.16 environment.
Upgrade to 1.18.0 using --upload-tools -> success
Upgrade t0 1.18.3 using --upload-tools -> failure
- agent state is down
- agent version stays as 1.18.0.1

Same happens if initial environment is 1.17.4
If initial deployment is 1.18.0 and upgrade to 1.18.3 it works.

Logs say 1.18.3 tools have been downloaded but unpacking seems to fail.

machine-0: 2014-03-11 04:32:47 ERROR juju runner.go:209 worker: fatal "upgrader": must restart: an agent upgrade is available
machine-0: 2014-03-11 04:32:47 INFO juju.provisioner provisioner_task.go:129 Shutting down provisioner task machine-0
machine-0: 2014-03-11 04:32:47 ERROR juju runner.go:209 worker: fatal "api": must restart: an agent upgrade is available
machine-0: 2014-03-11 04:32:47 INFO juju runner.go:164 worker: runner is dying
machine-0: 2014-03-11 04:32:47 INFO juju.state.apiserver apiserver.go:135 [1] machine-0 API connection terminated after 1.476748205s
machine-0: 2014-03-11 04:32:47 INFO juju.worker.upgrader error.go:32 upgraded from 1.18.0.1-precise-amd64 to 1.18.3.1-precise-amd64 ("https://s3.amazonaws.com/ea40acbee4a84e0e8009ed810d4e4679/tools/releases/juju-1.18.3.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJILHDJBMQGLFWX3A&Expires=1710131479&Signature=eLZCI7EsbLtgHHNFbkdhsBWc3ZA%3D")
machine-0: 2014-03-11 04:32:47 ERROR juju.cmd supercommand.go:296 must restart: an agent upgrade is available
machine-0: 2014-03-11 04:32:48 INFO juju.cmd.jujud machine.go:121 machine agent machine-0 start (1.18.0.1-precise-amd64)
machine-0: 2014-03-11 04:32:48 DEBUG juju.agent agent.go:315 read agent config, format "1.18"
machine-0: 2014-03-11 04:32:48 INFO juju.cmd.jujud machine.go:149 Starting StateWorker for machine-0
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "state"
machine-0: 2014-03-11 04:32:48 INFO juju.state open.go:79 opening state; mongo addresses: ["localhost:37017"]; entity "machine-0"
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "api"
machine-0: 2014-03-11 04:32:48 INFO juju apiclient.go:113 state/api: dialing "wss://localhost:17070/"
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "termination"
machine-0: 2014-03-11 04:32:48 ERROR juju apiclient.go:118 state/api: websocket.Dial wss://localhost:17070/: dial tcp 127.0.0.1:17070: connection refused
machine-0: 2014-03-11 04:32:48 ERROR juju runner.go:220 worker: exited "api": websocket.Dial wss://localhost:17070/: dial tcp 127.0.0.1:17070: connection refused
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:254 worker: restarting "api" in 3s
machine-0: 2014-03-11 04:32:48 INFO juju.state open.go:117 connection established
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "instancepoller"
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "apiserver"
machine-0: 2014-03-11 04:32:48 INFO juju.state.apiserver apiserver.go:43 listening on "[::]:17070"
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "cleaner"
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "resumer"
machine-0: 2014-03-11 04:32:48 INFO juju runner.go:262 worker: start "minunitsworker"
machine-0: 2014-03-11 04:32:51 INFO juju runner.go:262 worker: start "api"
machine-0: 2014-03-11 04:32:51 INFO juju apiclient.go:113 state/api: dialing "wss://localhost:17070/"
machine-0: 2014-03-11 04:32:51 INFO juju.state.apiserver apiserver.go:131 [1] API connection from 127.0.0.1:38833
machine-0: 2014-03-11 04:32:51 INFO juju apiclient.go:123 state/api: connection established
machine-0: 2014-03-11 04:32:51 DEBUG juju.state.apiserver apiserver.go:120 <- [1] <unknown> {"RequestId":1,"Type":"Admin","Request":"Login","Params":{"AuthTag":"machine-0","Password":"JyIyO9uoEWz2RVqLeCxKiD/i","Nonce":"user-admin:bootstrap"}}
machine-0: 2014-03-11 04:32:51 DEBUG juju.state.apiserver apiserver.go:127 -> [1] machine-0 2.46804ms {"RequestId":1,"Response":{}} Admin[""].Login
machine-0: 2014-03-11 04:32:51 DEBUG juju.state.apiserver apiserver.go:120 <- [1] machine-0 {"RequestId":2,"Type":"Agent","Request":"GetEntities","Params":{"Entities":[{"Tag":"machine-0"}]}}
machine-0: 2014-03-11 04:32:51 DEBUG juju.state.apiserver apiserver.go:127 -> [1] machine-0 561.558us {"RequestId":2,"Response":{"Entities":[{"Life":"alive","Jobs":["JobManageEnviron","JobHostUnits"],"ContainerType":"","Error":null}]}} Agent[""].GetEntities
machine-0: 2014-03-11 04:32:51 INFO juju runner.go:262 worker: start "upgrader"
machine-0: 2014-03-11 04:32:51 INFO juju runner.go:262 worker: start "upgrade-steps"
machine-0: 2014-03-11 04:32:51 INFO juju runner.go:262 worker: start "machiner"
machine-0: 2014-03-11 04:32:51 INFO juju.cmd.jujud machine.go:449 Upgrade to 1.18.0.1-precise-amd64 already completed.
machine-0: 2014-03-11 04:32:51 INFO juju.cmd.jujud machine.go:436 Upgrade to 1.18.0.1-precise-amd64 completed.

[Ian Booth (wallyworld)]

Looks like the unpacking of the new tools is failing because the destination of the untar is u.dataDir, but the agent config file has an empty value for dataDir.

ubuntu@ip-10-11-143-142:/var/lib/juju/agents/machine-0$ sudo cat agent.conf
# format 1.18
tag: machine-0
datadir: ""
logdir: ""
nonce: user-admin:bootstrap
upgradedToVersion: 1.18.0.1