socket.timeout as part of container.exec inside a pebble-ready hook

[Tom Haddon]

This pastebin illustrates the problem best: https://paste.ubuntu.com/p/KzS4r2PVV2/

We have an indico-pebble-ready hook that's firing. We run `indico_env_config = self._get_indico_env_config(container)` per this line of code https://github.com/canonical/indico-operator/blob/main/src/charm.py#L221. That then runs `process = container.exec(["indico", "setup", "list-plugins"])` per https://github.com/canonical/indico-operator/blob/main/src/charm.py#L338 but the `process.wait_output()` on the following line is resulting in a socket.timeout per the paste.

[Tom Haddon (mthaddon)]

Also just to mention that paste was from indico-pebble-ready, but here's a similar one just after from indico-celery-pebble-ready https://paste.ubuntu.com/p/Sn8NSC82Kj/. We've seen 5 container restarts on the indico-celery container since the pod was created as part of a charm upgrade, and 5 container restarts on indico-nginx. It's not clear that the `socket.timeout` errors are directly tied to the container restarts given we've also seen them (per the paste above) on containers that haven't restarted, but it does seem like it could be related, and certainly seems worth investigation.

[John A Meinel (jameinel)]

Also filed: https://github.com/canonical/operator/issues/826
With a lot of digging details about how I think the Operator Framework is running during this.
(I think this is a "pebble is under too much load, and OF should handle socket.timeout better") bug.