Too many database connections
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
Critical
|
Gavin Panella |
Bug Description
2015-08-24 21:20:56 [-] Unhandled failure dispatching AMP command. This
is probably a bug. Please ensure that this error is handled within
application code or declared in the signature of the SendEventMACAddress
command. [trusty-
Traceback (most recent call last):
File ".../threading.py", line 783, in __bootstrap
self.
File ".../threading.py", line 810, in __bootstrap_inner
self.run()
File ".../threading.py", line 763, in run
self.
--- <exception caught here> ---
File ".../twisted/
result = context.call(ctx, function, *args, **kwargs)
File ".../twisted/
return self.currentCon
File ".../twisted/
return func(*args,**kw)
File ".../provisioni
return func(*args, **kwargs)
File ".../maasserver
line 473, in call_within_
with connected(), post_commit_hooks:
File ".../contextlib
return self.gen.next()
File ".../maasserver
connectio
File ".../django/
line 124, in ensure_connection
self.
File ".../django/
six.
File ".../django/
line 124, in ensure_connection
self.
File ".../django/
self.
File ".../django/
line 116, in get_new_connection
return Database.
File ".../psycopg2/
connectio
django.
slots are reserved for non-replication superuser connections
It appears we need to limit database concurrency from within MAAS, or we
need to increase the number of connections permitted. Concurrency is
already somewhat limited because we have three size-limited threadpools.
However, each of those threadpools can contain up to 20 threads, giving
a total of 60 threads per process. We typically start 4 regiond
processes, giving a total of 240 threads. Each thread can have zero or
one connection.
In other words with everything running full tilt it is possible that
MAAS might attempt to hold 240 connections to the database at the same
time. This seems a little excessive.
In MAAS, the use of a thread almost always implies the use of a database
connection, hence a reasonable first approach would be to reduce the
#threads in each threadpool. However, we've seen that this can lead to
deadlock in other areas. We need more information.
description: | updated |
Changed in maas: | |
assignee: | nobody → Gavin Panella (allenap) |
status: | Triaged → In Progress |
Changed in maas: | |
milestone: | none → 1.9.0 |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas: | |
status: | Fix Committed → Fix Released |
2015-09-16 21:37:03 [HTTPChannel, 3,127.0. 0.1] 500 Error - /MAAS/api/ 1.0/pxeconfig/ python2. 7/dist- packages/ twisted/ web/wsgi. py", line 315, in run
appIterato r = self.applicatio n(self. environ, self.startResponse) python2. 7/dist- packages/ django/ core/handlers/ wsgi.py" , line 206, in __call__
response = self.get_ response( request) python2. 7/dist- packages/ maasserver/ utils/views. py", line 236, in get_response
response = get_response( request) python2. 7/dist- packages/ maasserver/ utils/views. py", line 225, in get_response
request, get_resolver(None), sys.exc_info()) python2. 7/dist- packages/ maasserver/ utils/views. py", line 209, in get_response atomic( ): python2. 7/dist- packages/ django/ db/transaction. py", line 237, in __enter__ get_autocommit( ): python2. 7/dist- packages/ django/ db/backends/ __init_ _.py", line 324, in get_autocommit
self. ensure_ connection( ) python2. 7/dist- packages/ django/ db/backends/ __init_ _.py", line 124, in ensure_connection
self. connect( ) python2. 7/dist- packages/ django/ db/utils. py", line 99, in __exit__
six. reraise( dj_exc_ type, dj_exc_value, traceback) python2. 7/dist- packages/ django/ db/backends/ __init_ _.py", line 124, in ensure_connection
self. connect( ) python2. 7/dist- packages/ django/ db/backends/ __init_ _.py", line 112, in connect
self. connection = self.get_ new_connection( conn_params) python2. 7/dist- packages/ django/ db/backends/ postgresql_ psycopg2/ base.py" , line 116, in get_new_connection connect( **conn_ params) python2. 7/dist- packages/ psycopg2/ __init_ _.py", line 179, in connect
connection _factory= connection_ factory, async=async)
django. db.utils. OperationalErro r: FATAL: remaining connection slots are reserved for non-replication superuser connections
Traceback (most recent call last):
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
--- <exception caught here> ---
File "/usr/lib/
with transaction.
File "/usr/lib/
if not connection.
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
return Database.
File "/usr/lib/