Application fails to connect to controller during install, connection reset by peer

[Alexander Balderson]

There are instances of a subordinate application is failing to connect to the juju controller during install and shutting down:

$ cat unit-telegraf-18.log | grep ERROR
2020-10-03 16:16:31 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 10.244.40.207:37098->10.244.40.114:17070: read: connection reset by peer
2020-10-03 16:16:31 DEBUG install ERROR codec.ReadHeader error: error receiving message: read tcp 10.244.40.207:37098->10.244.40.114:17070: read: connection reset by peer
2020-10-03 16:16:31 ERROR juju.worker.dependency engine.go:671 "proxy-config-updater" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 10.244.40.207:37098->10.244.40.114:17070: read: connection reset by peer
2020-10-03 16:16:31 ERROR juju.worker.dependency engine.go:671 "api-address-updater" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 10.244.40.207:37098->10.244.40.114:17070: read: connection reset by peer
2020-10-03 16:16:31 ERROR juju.worker.dependency engine.go:671 "migration-inactive-flag" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 10.244.40.207:37098->10.244.40.114:17070: read: connection reset by peer
2020-10-03 16:16:31 ERROR juju.worker.dependency engine.go:671 "migration-minion" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 10.244.40.207:37098->10.244.40.114:17070: read: connection reset by peer
2020-10-03 16:16:31 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 10.244.40.207:37098->10.244.40.114:17070: read: connection reset by peer
2020-10-03 16:16:35 DEBUG install ERROR connection is shut down
2020-10-03 16:16:44 DEBUG install ERROR connection is shut down
2020-10-03 16:16:44 ERROR juju.worker.uniter.context context.go:1146 cannot apply changes: connection is shut down
2020-10-03 16:16:44 ERROR juju.worker.uniter.operation runhook.go:136 hook "install" (via explicit, bespoke hook script) failed: cannot apply changes: connection is shut down

It is a subordinate application relating to an application in one model for placement, and relaying information to a 2nd model by way of CMR.

Full logs can be found at:
https://solutions.qa.canonical.com/openstack/testRun/37647399-9f15-4c08-a99b-0e48004b89cb

[Alexander Balderson (asbalderson)]

All future test runs which hit this bug can be found at
https://solutions.qa.canonical.com/bugs/bugs/bug/1898576

[Pete Vander Giessen (petevg)]

Triaged as medium and dropped into 2.8.6 milestone. This looks like the sort of sporadic issue that we should fix so that our Solutions QA tests will run more consistently.