Too many database connections

[Gavin Panella]

2015-08-24 21:20:56 [-] Unhandled failure dispatching AMP command. This
is probably a bug. Please ensure that this error is handled within
application code or declared in the signature of the SendEventMACAddress
command. [trusty-maas9:pid=8715:cmd=SendEventMACAddress:ask=197]
  Traceback (most recent call last):
    File ".../threading.py", line 783, in __bootstrap
      self.__bootstrap_inner()
    File ".../threading.py", line 810, in __bootstrap_inner
      self.run()
    File ".../threading.py", line 763, in run
      self.__target(*self.__args, **self.__kwargs)
  --- <exception caught here> ---
    File ".../twisted/python/threadpool.py", line 191, in _worker
      result = context.call(ctx, function, *args, **kwargs)
    File ".../twisted/python/context.py", line 118, in callWithContext
      return self.currentContext().callWithContext(ctx, func, *args, **kw)
    File ".../twisted/python/context.py", line 81, in callWithContext
      return func(*args,**kw)
    File ".../provisioningserver/utils/twisted.py", line 158, in wrapper
      return func(*args, **kwargs)
    File ".../maasserver/utils/orm.py",
    line 473, in call_within_transaction
      with connected(), post_commit_hooks:
    File ".../contextlib.py", line 17, in __enter__
      return self.gen.next()
    File ".../maasserver/utils/orm.py", line 409, in connected
      connection.ensure_connection()
    File ".../django/db/backends/__init__.py",
    line 124, in ensure_connection
      self.connect()
    File ".../django/db/utils.py", line 99, in __exit__
      six.reraise(dj_exc_type, dj_exc_value, traceback)
    File ".../django/db/backends/__init__.py",
    line 124, in ensure_connection
      self.connect()
    File ".../django/db/backends/__init__.py", line 112, in connect
      self.connection = self.get_new_connection(conn_params)
    File ".../django/db/backends/postgresql_psycopg2/base.py",
    line 116, in get_new_connection
      return Database.connect(**conn_params)
    File ".../psycopg2/__init__.py", line 179, in connect
      connection_factory=connection_factory, async=async)
  django.db.utils.OperationalError: FATAL: remaining connection
  slots are reserved for non-replication superuser connections

It appears we need to limit database concurrency from within MAAS, or we
need to increase the number of connections permitted. Concurrency is
already somewhat limited because we have three size-limited threadpools.
However, each of those threadpools can contain up to 20 threads, giving
a total of 60 threads per process. We typically start 4 regiond
processes, giving a total of 240 threads. Each thread can have zero or
one connection.

In other words with everything running full tilt it is possible that
MAAS might attempt to hold 240 connections to the database at the same
time. This seems a little excessive.

In MAAS, the use of a thread almost always implies the use of a database
connection, hence a reasonable first approach would be to reduce the
#threads in each threadpool. However, we've seen that this can lead to
deadlock in other areas. We need more information.

[Andres Rodriguez (andreserl)]

2015-09-16 21:37:03 [HTTPChannel,3,127.0.0.1] 500 Error - /MAAS/api/1.0/pxeconfig/
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
            appIterator = self.application(self.environ, self.startResponse)
          File "/usr/lib/python2.7/dist-packages/django/core/handlers/wsgi.py", line 206, in __call__
            response = self.get_response(request)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/views.py", line 236, in get_response
            response = get_response(request)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/views.py", line 225, in get_response
            request, get_resolver(None), sys.exc_info())
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/views.py", line 209, in get_response
            with transaction.atomic():
          File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 237, in __enter__
            if not connection.get_autocommit():
          File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 324, in get_autocommit
            self.ensure_connection()
          File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
            self.connect()
          File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
            six.reraise(dj_exc_type, dj_exc_value, traceback)
          File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
            self.connect()
          File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 112, in connect
            self.connection = self.get_new_connection(conn_params)
          File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 116, in get_new_connection
            return Database.connect(**conn_params)
          File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
            connection_factory=connection_factory, async=async)
        django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections

[Andres Rodriguez (andreserl)]

2015-09-16 21:37:03 [-] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within appli
cation code or declared in the signature of the SendEventMACAddress command. [trusty-maas9:pid=29662:cmd=SendEventMACAddress:ask=93]
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 763, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 197, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 480, in call_within_transaction
            with connected(), post_commit_hooks:
          File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
            return self.gen.next()
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 416, in connected
            connection.ensure_connection()
          File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
            self.connect()
          File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
            six.reraise(dj_exc_type, dj_exc_value, traceback)
          File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
            self.connect()
          File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 112, in connect
            self.connection = self.get_new_connection(conn_params)
          File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 116, in get_new_connection
            return Database.connect(**conn_params)
          File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
            connection_factory=connection_factory, async=async)
        django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections