Potential DB threadpool exhaustion issue in bootresources.py

[Jerzy Husakowski]

deferToDatabase() is meant to be used to run all DB-accessing code, and there's a threadpool (9 threads), each with own DB connection, to allow parallel queries. The code that is "deferredToDatabase" shouldn't call deferToDatabase() again, as it will grab another thread from the pool, and can deadlock in the worst case if there are no free threads left.

The following call sequence contains a deferToDatabase() call inside another deferToDatabase() call:
https://git.launchpad.net/maas/tree/src/maasserver/bootresources.py#n1455 calls _import_resources_in_thread(), which
https://git.launchpad.net/maas/tree/src/maasserver/bootresources.py#n1430 defers to database _import_resources(), which
https://git.launchpad.net/maas/tree/src/maasserver/bootresources.py#n1294 defers to database again.

[Jerzy Husakowski (jhusakowski)]

Let's introduce a test case to confirm whether the assumption that this will lead to a deadlock is correct, and re-triage this issue accordingly.

[Jacopo Rota (r00ta)]

I confirm this is a bug. Some ways to reproduce:

1) change `max_threads_for_database_pool = 1` in /src/maasserver/utils/threads.py
2) run `sudo maas init region+rack --database-uri maas-test-db:///`
3) createadmin
4) login

The import image page does not work

You could also
1) change `max_threads_for_database_pool = 1` in /src/maasserver/utils/threads.py
2) add some log debug statements in the deferred functions
3) login and make a POST call to `/MAAS/api/2.0/boot-resources/op-import` to trigger the deadlock

check the logs: the second deferred function is never called.

[Anton Troyanov (troyanov)]

Needs to be re-triaged for 3.5_ as we changed the way how we store image