Comment 2 for bug 1942804

Revision history for this message
Haw Loeung (hloeung) wrote :

It's trying to connect to the controllers via the advertised Fan addresses, but does eventually time out. Example snippet from a machine provisioned in Taipei using Juju controllers in PS5):

| 2021-09-09 08:40:07 ERROR juju.worker.dependency engine.go:671 "log-sender" manifold worker returned unexpected error: cannot send log message: websocket: close sent
| 2021-09-09 08:41:11 ERROR juju.worker.dependency engine.go:671 "migration-minion" manifold worker returned unexpected error: txn watcher sync error
| 2021-09-09 08:41:11 ERROR juju.worker.dependency engine.go:671 "log-sender" manifold worker returned unexpected error: cannot send log message: websocket: close sent
| 2021-09-09 08:41:11 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: api connection broken unexpectedly
| 2021-09-09 08:41:11 INFO juju.worker.logger logger.go:136 logger worker stopped
| 2021-09-09 08:41:17 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [093b6e] "machine-1" cannot open api: unable to connect to API: read tcp 211.75.xxx.xxx:37388->10.131.4.170:17070: read: connection reset by peer
| 2021-09-09 08:41:24 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [093b6e] "machine-1" cannot open api: unable to connect to API: dial tcp 252.170.0.1:17070: i/o timeout
| 2021-09-09 08:41:31 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [093b6e] "machine-1" cannot open api: unable to connect to API: dial tcp 252.109.0.1:17070: i/o timeout
| 2021-09-09 08:41:40 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [093b6e] "machine-1" cannot open api: unable to connect to API: dial tcp 252.170.0.1:17070: i/o timeout
| 2021-09-09 08:41:49 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [093b6e] "machine-1" cannot open api: unable to connect to API: dial tcp 252.109.0.1:17070: i/o timeout
| 2021-09-09 08:41:59 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [093b6e] "machine-1" cannot open api: unable to connect to API: dial tcp 252.109.0.1:17070: i/o timeout
| 2021-09-09 08:42:10 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [093b6e] "machine-1" cannot open api: unable to connect to API: dial tcp 252.109.0.1:17070: i/o timeout
| 2021-09-09 08:42:22 INFO juju.api apiclient.go:639 connection established to "wss://10.131.4.170:17070/model/093b6e8e-e00d-4c85-8052-52789d9e0132/api"
| 2021-09-09 08:42:22 INFO juju.worker.apicaller connect.go:158 [093b6e] "machine-1" successfully connected to "10.131.4.170:17070"
| 2021-09-09 08:42:25 INFO juju.worker.logger logger.go:120 logger worker started

(we're also doing something weird with DNAT'ing 10.xxx.xxx.xxx that the Juju controllers are using due to our weird and complex set up, this will change once we get routing between the two DCs fixed, but that's separate to this)