Deploying multiple model and applications causes juju to crash

[Simon Richardson]

When using tip of main (4.0) branch and attempting to deploy a lot of things quickly, it will fall over and crash.

I would expect the following to work, even if it's forced to slow down:

    juju bootstrap lxd test
    for i in $(seq 1 20); do juju add-model "model-$i" && (juju deploy ubuntu &); done

What actually happens is that we'll loose the leader and then it stalls, causing many dependency engine restarts. After a time, it will come back again and repeat the same scenario.

We should investigate if there is a way to mitigate this collapse, by either rate limiting the amount of work we can do, or pushing into a queue.

Note: Creating 40 models without deployment is easily achievable, so it's not the creation of the databases that is a problem.

   for i in $(seq 1 20); do juju add-model "model-$i"; done

A follow up for this is to add tracing (parca and OTEL) to see where we actually fall over.