I managed to reproduce it twice, so seems like reproducing the issue bootstraping from 2.1 branch and enabling HA is fairly having the issue:
machine-1: 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:21:51 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
Certificate misses DNS alternative names: ... X509v3 Subject Key Identifier: 9E:6A:52:5F:AE:17:AB:A6:C9:94:EA:82:B5:6C:19:52:0E:BD:4D:49 X509v3 Authority Key Identifier: keyid:E8:E4:03:62:97:36:19:39:4A:63:6B:18:A8:97:61:FC:C9:23:9C:C3
Signature Algorithm: sha256WithRSAEncryption ... machine-2:
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:20:49 ERROR juju.worker runner.go:199 fatal "2-container-watcher": connection is shut down 2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down 2017-08-16 09:22:00 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused 2017-08-16 09:22:00 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused 2017-08-16 09:22:01 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused 2017-08-16 09:22:02 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused 2017-08-16 09:22:02 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused 2017-08-16 09:22:03 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
Signature Algorithm: sha256WithRSAEncryption ...
I managed to reproduce it twice, so seems like reproducing the issue bootstraping from 2.1
branch and enabling HA is fairly having the issue:
machine-1:
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:21:51 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
Certificate misses DNS alternative names:
9E:6A: 52:5F:AE: 17:AB:A6: C9:94:EA: 82:B5:6C: 19:52:0E: BD:4D:49
keyid: E8:E4:03: 62:97:36: 19:39:4A: 63:6B:18: A8:97:61: FC:C9:23: 9C:C3
...
X509v3 Subject Key Identifier:
X509v3 Authority Key Identifier:
Signature Algorithm: sha256WithRSAEn cryption
...
machine-2:
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down watcher" : connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:20:49 ERROR juju.worker runner.go:199 fatal "2-container-
2017-08-16 09:20:49 ERROR juju.api.watcher watcher.go:87 error trying to stop watcher: connection is shut down
2017-08-16 09:22:00 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
2017-08-16 09:22:00 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
2017-08-16 09:22:01 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
2017-08-16 09:22:02 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
2017-08-16 09:22:02 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
2017-08-16 09:22:03 WARNING juju.mongo open.go:133 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: getsockopt: connection refused
Certificate misses DNS alternative names:
9E:6A: 52:5F:AE: 17:AB:A6: C9:94:EA: 82:B5:6C: 19:52:0E: BD:4D:49
keyid: E8:E4:03: 62:97:36: 19:39:4A: 63:6B:18: A8:97:61: FC:C9:23: 9C:C3
...
X509v3 Subject Key Identifier:
X509v3 Authority Key Identifier:
Signature Algorithm: sha256WithRSAEn cryption
...