juju.worker.httpserver worker.go:168 shutting down HTTP server

[Haw Loeung]

Hi,

One of the Juju controllers automatically shutdown the HTTP/API server. In the logs:

| 2020-06-20 15:30:05 ERROR juju.state.presence pruner.go:93 error removing old entries from "presence.pings": read tcp 192.168.16.5:52098->192.168.16.5:37017: i/o timeout
| 2020-06-20 15:30:05 WARNING juju.state.presence presence.go:380 error while pruning "presence.beings" for "be64b1c2-8da8-4964-87ac-567e90777877": read tcp 192.168.16.5:52098->192.168.16.5:37017: i/o timeout
| 2020-06-20 15:30:05 ERROR juju.worker.dependency engine.go:671 "state" manifold worker returned unexpected error: state ping failed: read tcp 192.168.16.5:52098->192.168.16.5:37017: i/o timeout
| 2020-06-20 15:30:05 ERROR juju.apiserver pubsub.go:96 pubsub receive error: read tcp 192.168.16.5:17070->192.168.16.6:54652: use of closed network connection
| 2020-06-20 15:30:05 ERROR juju.apiserver pubsub.go:96 pubsub receive error: read tcp 192.168.16.5:17070->192.168.16.4:51684: use of closed network connection
| 2020-06-20 15:30:05 INFO juju.apiserver.connection request_notifier.go:125 agent disconnected: machine-0 for f8acb894-c0cf-4360-8f44-d7444687823b
| 2020-06-20 15:30:05 INFO juju.apiserver.connection request_notifier.go:125 agent disconnected: machine-2 for f8acb894-c0cf-4360-8f44-d7444687823b
| 2020-06-20 15:30:05 ERROR juju.worker.dependency engine.go:671 "log-sender" manifold worker returned unexpected error: cannot send log message: write tcp 127.0.0.1:58802->127.0.0.1:17070: write: broken pipe
| ...
| 2020-06-20 15:31:05 WARNING juju.worker.httpserver worker.go:210 timeout waiting for apiserver shutdown
| 2020-06-20 15:31:05 INFO juju.worker.httpserver worker.go:168 shutting down HTTP server

It started itself up shortly after:

| 2020-06-20 15:31:05 INFO juju.worker.httpserver worker.go:293 listening on "[::]:17070"

Output from apiserver-debug.log:

| https://pastebin.ubuntu.com/p/KbnmpbYKFF/

This is running Juju 2.7.6. Let me know if you want to see the machine-1.log entries for that time.

[Pen Gale (pengale)]

It looks like this is Juju running as expected -- the http server ran into an error, and so the worker restarted it.

Did this cause other issues during a deployment or test?

[Tim Penhey (thumper)]

This is the root cause:

2020-06-20 15:30:05 ERROR juju.worker.dependency engine.go:671 "state" manifold worker returned unexpected error: state ping failed: read tcp 192.168.16.5:52098->192.168.16.5:37017: i/o timeout

If the database doesn't respond to pings, the state worker is restarted, which restarts most of the internal workers, apiserver and httpserver included.

[Haw Loeung (hloeung)]

How many pings and over what period of time before it is configured "database not responding to pings" ?

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Joseph Phillips (manadart)]

The model state worker will ping the database every 15 seconds.

An error is returned (causing the state worker restart) if any such pings fail.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]