Agents failing, blocked by upgrade but no upgrade performed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Invalid
|
Undecided
|
Unassigned | ||
juju-core |
Expired
|
Undecided
|
Unassigned | ||
1.25 |
Expired
|
Undecided
|
Unassigned |
Bug Description
With 1.25.6, on both lxd and OpenStack, I'm ending up with agents on units going into a failed state. Investigating the machine-*.log files on a failed unit, I find this at the end:
2016-08-23 08:28:07 ERROR juju.worker runner.go:223 exited "api": login for "machine-16" blocked because upgrade in progress
Note that no upgrade was performed. This was a recently deployed environment, deployed with 1.25.6. The unit was fine and last know to have reconfigured its logging, happily deploying on a very slow OpenStack but 20 mins later (08:24) failed. It appears machine 0 is deciding to upgrade itself from 1.25.6 to 1.25.6:
2016-08-23 08:04:35 DEBUG juju.network network.go:268 no lxc bridge addresses to filter for machine
2016-08-23 08:04:35 INFO juju.worker.
2016-08-23 08:04:35 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "upgrader": connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "16-container-
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "apiaddressupda
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "proxyupdater": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "deployer": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "stateconverter": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "logger": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "authentication
2016-08-23 08:24:06 ERROR juju.worker.
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "machiner": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "reboot": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "rsyslog": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "storageprovisi
2016-08-23 08:24:06 ERROR juju.worker runner.go:212 fatal "api-post-upgrade": connection is shut down
2016-08-23 08:24:06 ERROR juju.worker runner.go:223 exited "api": connection is shut down
2016-08-23 08:24:09 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:24:12 ERROR juju.worker runner.go:223 exited "api": login for "machine-16" blocked because upgrade in progress
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "proxyupdater": connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "apiaddressupda
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "upgrader": connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "rsyslog": connection is shut down
2016-08-23 08:27:04 ERROR juju.worker.
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "storageprovisi
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "authentication
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "machiner": connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "logger": connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "stateconverter": connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "16-container-
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "deployer": connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "reboot": connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:212 fatal "api-post-upgrade": connection is shut down
2016-08-23 08:27:04 ERROR juju.worker runner.go:223 exited "api": connection is shut down
2016-08-23 08:27:07 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:10 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:13 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:16 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:19 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:22 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:25 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:28 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:31 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:34 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:37 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:40 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:43 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:46 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:49 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:52 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:55 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:27:58 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:28:01 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:28:04 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:28:07 ERROR juju.worker runner.go:223 exited "api": login for "machine-16" blocked because upgrade in progress
And on machine-0:
2016-08-23 08:03:31 WARNING juju.apiserver.
2016-08-23 08:24:07 INFO juju.cmd supercommand.go:37 running jujud [1.25.6-
2016-08-23 08:24:07 DEBUG juju.agent agent.go:491 read agent config, format "1.18"
2016-08-23 08:24:07 INFO juju.cmd.jujud machine.go:419 machine agent machine-0 start (1.25.6-
2016-08-23 08:24:07 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/
ile or directory
2016-08-23 08:24:07 INFO juju.cmd.jujud upgrade.go:88 no upgrade steps required or upgrade steps for 1.25.6 have already
been run.
2016-08-23 08:24:07 INFO juju.network network.go:242 setting prefer-ipv6 to false
2016-08-23 08:24:07 INFO juju.worker runner.go:269 start "api"
2016-08-23 08:24:07 DEBUG juju.worker.
2016-08-23 08:24:07 INFO juju.worker runner.go:269 start "statestarter"
2016-08-23 08:24:07 DEBUG juju.worker runner.go:196 "statestarter" started
2016-08-23 08:24:07 INFO juju.api apiclient.go:262 dialing "wss://
-73e9b5e500f7/api"
2016-08-23 08:24:07 INFO juju.worker runner.go:269 start "termination"
2016-08-23 08:24:07 DEBUG juju.worker runner.go:196 "termination" started
2016-08-23 08:24:07 INFO juju.worker runner.go:269 start "state"
2016-08-23 08:24:07 INFO juju.api apiclient.go:262 dialing "wss://
e9b5e500f7/api"
2016-08-23 08:24:07 INFO juju.api apiclient.go:270 error dialing "wss://
9-8adf-
dial tcp 10.55.32.252:17070: connection refused
2016-08-23 08:24:07 INFO juju.api apiclient.go:270 error dialing "wss://
adf-73e9b5e500f
cp 127.0.0.1:17070: connection refused
2016-08-23 08:24:07 DEBUG juju.worker.
2016-08-23 08:24:07 INFO juju.worker runner.go:275 stopped "api", err: unable to connect to "wss://
onment/
2016-08-23 08:24:07 DEBUG juju.worker runner.go:203 "api" done: unable to connect to "wss://
d0720df8-
2016-08-23 08:24:07 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
t/d0720df8-
2016-08-23 08:24:07 INFO juju.worker runner.go:261 restarting "api" in 3s
2016-08-23 08:24:07 DEBUG juju.service discovery.go:115 discovered init system "upstart" from local host
2016-08-23 08:24:07 DEBUG juju.cmd.jujud machine.go:1409 mongodb service is installed
2016-08-23 08:24:07 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:07 INFO juju.mongo open.go:125 dialled mongo successfully on address "[::1]:37017"
2016-08-23 08:24:07 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:07 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:07 DEBUG juju.cmd.jujud machine.go:1569 replicaset member count: 1
2016-08-23 08:24:07 INFO juju.mongo mongo.go:208 Ensuring mongo server is running; data directory /var/lib/juju; port 37017
2016-08-23 08:24:07 INFO juju.mongo mongo.go:372 installing juju-mongodb
2016-08-23 08:24:07 INFO juju.utils.
2016-08-23 08:24:08 DEBUG juju.mongo mongo.go:331 using mongod: /usr/lib/
2016-08-23 08:24:08 DEBUG juju.service discovery.go:115 discovered init system "upstart" from local host
2016-08-23 08:24:08 DEBUG juju.mongo mongo.go:276 mongo exists as expected
2016-08-23 08:24:08 INFO juju.state open.go:51 opening state, mongo addresses: ["127.0.
2016-08-23 08:24:08 DEBUG juju.state open.go:52 dialing mongo
2016-08-23 08:24:08 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:08 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:08 DEBUG juju.state open.go:57 connection established
2016-08-23 08:24:08 DEBUG juju.state open.go:64 mongodb login successful
2016-08-23 08:24:09 INFO juju.state state.go:195 creating lease client as machine-0
2016-08-23 08:24:09 INFO juju.state state.go:208 starting leadership manager
2016-08-23 08:24:09 INFO juju.state state.go:219 creating cloud image metadata storage
2016-08-23 08:24:09 INFO juju.state state.go:222 starting presence watcher
2016-08-23 08:24:09 DEBUG juju.state.
2016-08-23 08:24:09 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:09 DEBUG juju.environs imagemetadata.go:46 new user image datasource registered: environment storage
2016-08-23 08:24:09 INFO juju.worker.
2016-08-23 08:24:09 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 1
2016-08-23 08:24:09 INFO juju.worker runner.go:269 start "env worker manager"
2016-08-23 08:24:09 INFO juju.worker runner.go:269 start "peergrouper"
2016-08-23 08:24:09 DEBUG juju.worker runner.go:196 "env worker manager" started
2016-08-23 08:24:09 DEBUG juju.worker runner.go:196 "peergrouper" started
2016-08-23 08:24:09 INFO juju.worker runner.go:269 start "restore"
2016-08-23 08:24:09 DEBUG juju.worker runner.go:196 "restore" started
2016-08-23 08:24:09 INFO juju.worker runner.go:269 start "apiserver"
2016-08-23 08:24:09 DEBUG juju.cmd.jujud machine.go:1084 opening state for apistate worker
2016-08-23 08:24:09 INFO juju.state open.go:51 opening state, mongo addresses: ["127.0.
2016-08-23 08:24:09 DEBUG juju.state open.go:52 dialing mongo
2016-08-23 08:24:09 INFO juju.worker.
2016-08-23 08:24:09 INFO juju.worker.
2016-08-23 08:24:09 INFO juju.worker runner.go:269 start "certupdater"
2016-08-23 08:24:09 DEBUG juju.worker runner.go:196 "certupdater" started
2016-08-23 08:24:09 INFO juju.worker runner.go:269 start "statushistoryp
2016-08-23 08:24:09 INFO juju.worker runner.go:269 start "txnpruner"
2016-08-23 08:24:09 DEBUG juju.worker runner.go:196 "statushistoryp
2016-08-23 08:24:09 DEBUG juju.worker runner.go:196 "txnpruner" started
2016-08-23 08:24:09 DEBUG juju.worker runner.go:196 "state" started
2016-08-23 08:24:09 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:09 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:09 DEBUG juju.state open.go:57 connection established
2016-08-23 08:24:09 DEBUG juju.state open.go:64 mongodb login successful
2016-08-23 08:24:10 INFO juju.worker runner.go:269 start "api"
2016-08-23 08:24:10 DEBUG juju.worker.
2016-08-23 08:24:10 INFO juju.api apiclient.go:262 dialing "wss://
2016-08-23 08:24:10 INFO juju.api apiclient.go:270 error dialing "wss://
2016-08-23 08:24:10 INFO juju.api apiclient.go:262 dialing "wss://
2016-08-23 08:24:10 INFO juju.api apiclient.go:270 error dialing "wss://
2016-08-23 08:24:10 DEBUG juju.worker.
2016-08-23 08:24:10 INFO juju.worker runner.go:275 stopped "api", err: unable to connect to "wss://
2016-08-23 08:24:10 DEBUG juju.worker runner.go:203 "api" done: unable to connect to "wss://
2016-08-23 08:24:10 ERROR juju.worker runner.go:223 exited "api": unable to connect to "wss://
2016-08-23 08:24:10 INFO juju.worker runner.go:261 restarting "api" in 3s
2016-08-23 08:24:10 INFO juju.state state.go:195 creating lease client as machine-0
2016-08-23 08:24:10 INFO juju.state state.go:208 starting leadership manager
2016-08-23 08:24:10 INFO juju.state state.go:219 creating cloud image metadata storage
2016-08-23 08:24:10 INFO juju.state state.go:222 starting presence watcher
2016-08-23 08:24:10 DEBUG juju.state.
2016-08-23 08:24:10 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:10 INFO juju.apiserver apiserver.go:168 listening on "[::]:17070"
2016-08-23 08:24:10 DEBUG juju.worker runner.go:196 "apiserver" started
2016-08-23 08:24:11 INFO juju.apiserver apiserver.go:281 [1] API connection from 10.55.32.241:54204
2016-08-23 08:24:11 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:262 <- [1] <unknown> {"RequestId"
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:276 -> [1] <unknown> 10.266059ms {"RequestId"
2016-08-23 08:24:12 INFO juju.apiserver apiserver.go:285 [1] <unknown> API connection terminated after 13.218289ms
2016-08-23 08:24:12 INFO juju.apiserver apiserver.go:281 [2] API connection from 10.55.32.248:45674
2016-08-23 08:24:12 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:262 <- [2] <unknown> {"RequestId"
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:276 -> [2] <unknown> 364.95us {"RequestId"
2016-08-23 08:24:12 INFO juju.apiserver apiserver.go:285 [2] <unknown> API connection terminated after 12.445585ms
2016-08-23 08:24:12 INFO juju.apiserver apiserver.go:281 [3] API connection from 10.55.32.237:55528
2016-08-23 08:24:12 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:262 <- [3] <unknown> {"RequestId"
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:276 -> [3] <unknown> 225.574us {"RequestId"
2016-08-23 08:24:12 INFO juju.apiserver apiserver.go:285 [3] <unknown> API connection terminated after 6.039444ms
2016-08-23 08:24:12 INFO juju.apiserver apiserver.go:281 [4] API connection from 10.55.32.233:38932
2016-08-23 08:24:12 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:262 <- [4] <unknown> {"RequestId"
2016-08-23 08:24:12 DEBUG juju.apiserver apiserver.go:276 -> [4] <unknown> 346.556us {"RequestId"
2016-08-23 08:24:12 INFO juju.apiserver apiserver.go:285 [4] <unknown> API connection terminated after 21.324366ms
2016-08-23 08:24:13 INFO juju.apiserver apiserver.go:281 [5] API connection from 10.172.254.15:37298
2016-08-23 08:24:13 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:13 INFO juju.worker runner.go:269 start "api"
2016-08-23 08:24:13 DEBUG juju.worker.
2016-08-23 08:24:13 INFO juju.api apiclient.go:262 dialing "wss://
2016-08-23 08:24:13 INFO juju.api apiclient.go:262 dialing "wss://
2016-08-23 08:24:13 INFO juju.apiserver apiserver.go:281 [6] API connection from 10.55.32.252:40949
2016-08-23 08:24:13 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:13 INFO juju.api apiclient.go:194 connection established to "wss://
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [6] <unknown> {"RequestId"
2016-08-23 08:24:13 INFO juju.apiserver apiserver.go:281 [7] API connection from 127.0.0.1:58066
2016-08-23 08:24:13 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:13 INFO juju.apiserver apiserver.go:285 [7] <unknown> API connection terminated after 1.57543ms
2016-08-23 08:24:13 DEBUG juju.apiserver admin.go:149 hostPorts: [[10.55.
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 71.787879ms {"RequestId"
2016-08-23 08:24:13 DEBUG juju.worker.
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [5] <unknown> {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 3.644944ms {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:13 INFO juju.apiserver apiserver.go:281 [8] API connection from 10.172.254.15:37296
2016-08-23 08:24:13 DEBUG juju.apiserver utils.go:71 validate env uuid: state server environment - d0720df8-
2016-08-23 08:24:13 DEBUG juju.apiserver admin.go:149 hostPorts: [[10.55.
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:276 -> [5] user-admin@local 75.392329ms {"RequestId"
2016-08-23 08:24:13 INFO juju.apiserver.
2016-08-23 08:24:13 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:13 INFO juju.apiserver.
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 310.646577ms {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 2.318127ms {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 1.893209ms {"RequestId"
2016-08-23 08:24:13 INFO juju.worker runner.go:269 start "upgrader"
2016-08-23 08:24:13 DEBUG juju.worker runner.go:196 "upgrader" started
2016-08-23 08:24:13 INFO juju.worker runner.go:269 start "upgrade-steps"
2016-08-23 08:24:13 INFO juju.worker runner.go:269 start "api-post-upgrade"
2016-08-23 08:24:13 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/
2016-08-23 08:24:13 DEBUG juju.worker runner.go:196 "upgrade-steps" started
2016-08-23 08:24:13 DEBUG juju.worker runner.go:196 "api-post-upgrade" started
2016-08-23 08:24:13 DEBUG juju.worker runner.go:196 "api" started
2016-08-23 08:24:13 INFO juju.worker runner.go:275 stopped "upgrade-steps", err: <nil>
2016-08-23 08:24:13 DEBUG juju.worker runner.go:203 "upgrade-steps" done: <nil>
2016-08-23 08:24:13 DEBUG juju.worker runner.go:206 removing "upgrade-steps" from known workers
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [8] <unknown> {"RequestId"
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:262 <- [5] user-admin@local {"RequestId"
2016-08-23 08:24:13 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:13 DEBUG juju.apiserver.
2016-08-23 08:24:13 DEBUG juju.apiserver admin.go:149 hostPorts: [[10.55.
2016-08-23 08:24:13 DEBUG juju.apiserver apiserver.go:276 -> [8] user-admin@local 93.489082ms {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [5] user-admin@local 92.513183ms {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 271.822371ms {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 2.853607ms {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 3.490731ms {"RequestId"
2016-08-23 08:24:14 INFO juju.worker.
2016-08-23 08:24:14 DEBUG juju.cmd.jujud machine.go:1667 upgrades done, starting worker "env worker manager"
2016-08-23 08:24:14 DEBUG juju.cmd.jujud machine.go:1667 upgrades done, starting worker "peergrouper"
2016-08-23 08:24:14 DEBUG juju.cmd.jujud machine.go:1667 upgrades done, starting worker "restore"
2016-08-23 08:24:14 DEBUG juju.cmd.jujud machine.go:1667 upgrades done, starting worker "certupdater"
2016-08-23 08:24:14 DEBUG juju.cmd.jujud machine.go:1667 upgrades done, starting worker "statushistoryp
2016-08-23 08:24:14 DEBUG juju.cmd.jujud machine.go:1667 upgrades done, starting worker "txnpruner"
2016-08-23 08:24:14 DEBUG juju.cmd.jujud machine.go:1667 upgrades done, starting worker "api-post-upgrade"
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "proxyupdater"
2016-08-23 08:24:14 DEBUG juju.worker.
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "proxyupdater" started
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "resumer"
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "resumer" started
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:14 DEBUG juju.worker.
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 63.988666ms {"RequestId"
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "d0720df8-
2016-08-23 08:24:14 INFO juju.state open.go:51 opening state, mongo addresses: ["127.0.
2016-08-23 08:24:14 DEBUG juju.state open.go:52 dialing mongo
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 51.67865ms {"RequestId"
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "machiner"
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "reboot"
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "machiner" started
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "reboot" started
2016-08-23 08:24:14 DEBUG juju.worker.reboot reboot.go:54 Reboot worker setup
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "apiaddressupdater"
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "apiaddressupdater" started
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "logger"
2016-08-23 08:24:14 DEBUG juju.worker.logger logger.go:35 initial log config: "<root>=DEBUG"
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "rsyslog"
2016-08-23 08:24:14 DEBUG juju.worker.rsyslog worker.go:108 starting rsyslog worker mode 2 for "machine-0" ""
2016-08-23 08:24:14 DEBUG juju.worker.logger logger.go:60 logger setup
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "logger" started
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "rsyslog" started
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "diskmanager"
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "diskmanager" started
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "storageprovisi
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "networker"
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "storageprovisi
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "networker" started
2016-08-23 08:24:14 INFO juju.networker networker.go:163 networker is disabled - not starting on machine "machine-0"
2016-08-23 08:24:14 INFO juju.worker runner.go:269 start "authentication
2016-08-23 08:24:14 DEBUG juju.worker.
2016-08-23 08:24:14 DEBUG juju.worker.
2016-08-23 08:24:14 INFO juju.worker runner.go:275 stopped "networker", err: <nil>
2016-08-23 08:24:14 DEBUG juju.worker runner.go:196 "authentication
2016-08-23 08:24:14 DEBUG juju.worker runner.go:203 "networker" done: <nil>
2016-08-23 08:24:14 DEBUG juju.worker runner.go:206 removing "networker" from known workers
2016-08-23 08:24:14 INFO juju.apiserver apiserver.go:285 [5] user-admin@local API connection terminated after 1.049061379s
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [8] user-admin@local {"RequestId"
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 100.661653ms {"RequestId"
2016-08-23 08:24:14 DEBUG juju.worker.
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 39.854443ms {"RequestId"
2016-08-23 08:24:14 DEBUG juju.worker.
2016-08-23 08:24:14 DEBUG juju.worker.
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:14 DEBUG juju.network network.go:268 no lxc bridge addresses to filter for machine
2016-08-23 08:24:14 INFO juju.worker.
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 63.895072ms {"RequestId"
2016-08-23 08:24:14 DEBUG juju.state open.go:57 connection established
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:276 -> [6] machine-0 66.734068ms {"RequestId"
2016-08-23 08:24:14 INFO juju.worker.
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:14 INFO juju.mongo open.go:125 dialled mongo successfully on address "10.55.
2016-08-23 08:24:14 DEBUG juju.container.kvm kvm.go:71 kvm-ok output:
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm_amd
INFO: Your CPU supports KVM extensions
KVM acceleration can be used
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.apiserver apiserver.go:262 <- [6] machine-0 {"RequestId"
2016-08-23 08:24:14 DEBUG juju.state open.go:64 mongodb login successful
2016-08-23 08:24:14 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>
2016-08-23 08:28:02 INFO juju.cmd supercommand.go:37 running jujud [1.25.6-
2016-08-23 08:28:02 DEBUG juju.agent agent.go:491 read agent config, format "1.18"
2016-08-23 08:28:02 INFO juju.cmd.jujud machine.go:419 machine agent machine-0 start (1.25.6-
2016-08-23 08:28:02 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/
2016-08-23 08:28:02 INFO juju.cmd.jujud upgrade.go:88 no upgrade steps required or upgrade steps for 1.25.6 have already been run.
tags: | added: canonical-is |
affects: | juju → juju-core |
no longer affects: | juju-core |
Changed in juju-core: | |
status: | Incomplete → Triaged |
importance: | Undecided → Critical |
status: | Triaged → Invalid |
importance: | Critical → Undecided |
Changed in juju-core: | |
status: | Invalid → Triaged |
importance: | Undecided → Critical |
Changed in juju-core: | |
milestone: | none → 1.25.11 |
I haven't been able to reproduce this after downgrading and forcing 1.25.5