Provisioner worker pool errors cause on-machine provisioning to cease

[Joseph Phillips]

The on-machine provisioner for LXD/KVM uses a worker pool implemented using worker.Runner.

It is possible for errors encountered by the runners to be interpreted as fatal, which shuts down the worker pool without ever restarting it.

When this happens, container provisioning is effectively halted on the machine until the jujud daemon is restarted.

We should in most cases, not interpret provisioner task errors as fatal.

An example can be seen here, where a transient API connection error causes the fatal error:
https://pastebin.ubuntu.com/p/gYhRmNMRNZ/

[Joseph Phillips (manadart)]

This happens because the container provisioner is not started via dependency engine, rather by the machine agent directly.

So the normal mechanism whereby a worker would be restarted when its dependencies bounce due to an error, is not in play.

[Joseph Phillips (manadart)]

Actually it is (by proxy) tied to the dependency engine. It is run via APIWorkersManifold. It should be restarted if the APICaller bounces.

[Heather Lanigan (hmlanigan)]

Working on moving functionality of the unconverted-api-workers into their own workers and manifolds. They are causing multiple issues around shutting down machine agent and preventing migration in their current form.

[Heather Lanigan (hmlanigan)]

https://github.com/juju/juju/pull/14834