We are trying to deploy juju HA on customer site.
With same scenario,
$ juju bootstrap
$ juju ensure-availability
Then kill mongo on machine 0 (master state server).
Juju stop working. (juju status not respond.) We think this is a bug for HA.
===== juju log ============
ubuntu@maas-trusty:~$ juju status --debug
2014-10-19 17:42:14 INFO juju.cmd supercommand.go:37 running juju [1.20.8-trusty-amd64 gc]
2014-10-19 17:42:14 DEBUG juju.conn api.go:187 trying cached API connection settings
2014-10-19 17:42:14 INFO juju.conn api.go:270 connecting to API addresses: [bootstrap-trusty-01.beijing.cts.canonical.com:17070 bootstrap-trusty-01.beijing.cts.canonical.com:17070 10.231.64.39:17070 bootstrap-trusty-02.beijing.cts.canonical.com:17070 bootstrap-trusty-02.beijing.cts.canonical.com:17070 10.231.64.87:17070 bootstrap-trusty-03.beijing.cts.canonical.com:17070 bootstrap-trusty-03.beijing.cts.canonical.com:17070 10.231.64.88:17070]
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-01.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-01.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://10.231.64.39:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-02.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-02.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://10.231.64.87:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 DEBUG juju.state.api apiclient.go:248 error dialing "wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api", will retry: websocket.Dial wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api: dial tcp 10.231.64.88:17070: connection refused
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 DEBUG juju.state.api apiclient.go:248 error dialing "wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api", will retry: websocket.Dial wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api: dial tcp 10.231.64.88:17070: connection refused
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://10.231.64.88:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:14 DEBUG juju.state.api apiclient.go:248 error dialing "wss://10.231.64.88:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api", will retry: websocket.Dial wss://10.231.64.88:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api: dial tcp 10.231.64.88:17070: connection refused
2014-10-19 17:42:16 DEBUG juju.provider.maas environprovider.go:30 opening environment "maas".
2014-10-19 17:42:16 DEBUG juju.provider.common state.go:83 waiting for addresses of state server instances [/MAAS/api/1.0/nodes/node-cf194ab4-072b-11e4-beb5-00163e0f4f3c/]
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-01.beijing.cts.canonical.com:17070/"
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-01.beijing.cts.canonical.com:17070/"
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
2014-10-19 17:42:16 DEBUG juju.state.api apiclient.go:248 error dialing "wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api", will retry: websocket.Dial wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api: dial tcp 10.231.64.88:17070: connection refused
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://10.231.64.39:17070/"
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://bootstrap-trusty-03.beijing.cts.canonical.com:17070/environment/cf1c570b-0611-4585-8915-fc3fb53024d1/api"
We are trying to deploy juju HA on customer site.
With same scenario,
$ juju bootstrap
$ juju ensure-availability
Then kill mongo on machine 0 (master state server).
Juju stop working. (juju status not respond.) We think this is a bug for HA.
===== juju log ============ maas-trusty: ~$ juju status --debug trusty- amd64 gc] trusty- 01.beijing. cts.canonical. com:17070 bootstrap- trusty- 01.beijing. cts.canonical. com:17070 10.231.64.39:17070 bootstrap- trusty- 02.beijing. cts.canonical. com:17070 bootstrap- trusty- 02.beijing. cts.canonical. com:17070 10.231.64.87:17070 bootstrap- trusty- 03.beijing. cts.canonical. com:17070 bootstrap- trusty- 03.beijing. cts.canonical. com:17070 10.231.64.88:17070] bootstrap- trusty- 01.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" bootstrap- trusty- 01.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" 10.231. 64.39:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" bootstrap- trusty- 02.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" bootstrap- trusty- 02.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" 10.231. 64.87:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" bootstrap- trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" bootstrap- trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" , will retry: websocket.Dial wss://bootstrap -trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api: dial tcp 10.231.64.88:17070: connection refused bootstrap- trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" bootstrap- trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" , will retry: websocket.Dial wss://bootstrap -trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api: dial tcp 10.231.64.88:17070: connection refused 10.231. 64.88:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" 10.231. 64.88:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" , will retry: websocket.Dial wss://10. 231.64. 88:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api: dial tcp 10.231.64.88:17070: connection refused .go:30 opening environment "maas". common state.go:83 waiting for addresses of state server instances [/MAAS/ api/1.0/ nodes/node- cf194ab4- 072b-11e4- beb5-00163e0f4f 3c/] bootstrap- trusty- 01.beijing. cts.canonical. com:17070/ " bootstrap- trusty- 01.beijing. cts.canonical. com:17070/ " bootstrap- trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" bootstrap- trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api" , will retry: websocket.Dial wss://bootstrap -trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api: dial tcp 10.231.64.88:17070: connection refused 10.231. 64.39:17070/ " bootstrap- trusty- 03.beijing. cts.canonical. com:17070/ environment/ cf1c570b- 0611-4585- 8915-fc3fb53024 d1/api"
ubuntu@
2014-10-19 17:42:14 INFO juju.cmd supercommand.go:37 running juju [1.20.8-
2014-10-19 17:42:14 DEBUG juju.conn api.go:187 trying cached API connection settings
2014-10-19 17:42:14 INFO juju.conn api.go:270 connecting to API addresses: [bootstrap-
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 DEBUG juju.state.api apiclient.go:248 error dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 DEBUG juju.state.api apiclient.go:248 error dialing "wss://
2014-10-19 17:42:14 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:14 DEBUG juju.state.api apiclient.go:248 error dialing "wss://
2014-10-19 17:42:16 DEBUG juju.provider.maas environprovider
2014-10-19 17:42:16 DEBUG juju.provider.
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:16 DEBUG juju.state.api apiclient.go:248 error dialing "wss://
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://
2014-10-19 17:42:16 INFO juju.state.api apiclient.go:242 dialing "wss://