Activity log for bug #1799871

Date Who What changed Old value New value Message
2018-10-25 03:56:13 Andres Rodriguez bug added bug
2018-10-25 03:56:17 Andres Rodriguez maas: importance Undecided Critical
2018-10-25 03:56:19 Andres Rodriguez maas: status New Triaged
2018-10-25 03:56:25 Andres Rodriguez maas: assignee Blake Rouse (blake-rouse)
2018-10-25 03:56:26 Andres Rodriguez maas: milestone 2.5.0rc1
2018-10-25 03:56:53 Andres Rodriguez description 2018-10-25 03:55:05 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'GetSyslogConfiguration' command. [maas00:pid=12227:cmd=GetSyslogConfiguration:ask=bb38] Traceback (most recent call last): File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 267, in run self._asyncioEventloop.run_forever() File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run f(*args, **kwargs) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback self._startRunCallbacks(fail) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1171, in checkKnownErrors key = error.trap(*command.allErrors) File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap self.raiseException() File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException raise self.value.with_traceback(self.tb) File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext result = inContext.theWork() File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda> inContext.theWork = lambda: context.call(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw) File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper result = func(*args, **kwargs) File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 755, in call_within_transaction with connected(), post_commit_hooks: File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 684, in connected connection.ensure_connection() File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection self.connect() File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise raise value.with_traceback(tb) File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection self.connect() File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 189, in connect self.connection = self.get_new_connection(conn_params) File "/usr/lib/python3/dist-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection connection = Database.connect(**conn_params) File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 130, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections I had a MAAS that has 2 region/racks + 2 racks controllers. 2 physical machines, 1 deployed as a pod. All of the sudden I started seeing this issue. 2018-10-25 03:55:05 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'GetSyslogConfiguration' command. [maas00:pid=12227:cmd=GetSyslogConfiguration:ask=bb38]  Traceback (most recent call last):    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 267, in run      self._asyncioEventloop.run_forever()    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run      f(*args, **kwargs)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback      self._startRunCallbacks(fail)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks      self._runCallbacks()  --- <exception caught here> ---    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks      current.result = callback(current.result, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1171, in checkKnownErrors      key = error.trap(*command.allErrors)    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap      self.raiseException()    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException      raise self.value.with_traceback(self.tb)    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext      result = inContext.theWork()    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>      inContext.theWork = lambda: context.call(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext      return self.currentContext().callWithContext(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext      return func(*args,**kw)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext      return func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper      result = func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 755, in call_within_transaction      with connected(), post_commit_hooks:    File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__      return next(self.gen)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 684, in connected      connection.ensure_connection()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__      six.reraise(dj_exc_type, dj_exc_value, traceback)    File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise      raise value.with_traceback(tb)    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 189, in connect      self.connection = self.get_new_connection(conn_params)    File "/usr/lib/python3/dist-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection      connection = Database.connect(**conn_params)    File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 130, in connect      conn = _connect(dsn, connection_factory=connection_factory, **kwasync)  django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
2018-10-25 04:17:01 Andres Rodriguez description I had a MAAS that has 2 region/racks + 2 racks controllers. 2 physical machines, 1 deployed as a pod. All of the sudden I started seeing this issue. 2018-10-25 03:55:05 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'GetSyslogConfiguration' command. [maas00:pid=12227:cmd=GetSyslogConfiguration:ask=bb38]  Traceback (most recent call last):    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 267, in run      self._asyncioEventloop.run_forever()    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run      f(*args, **kwargs)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback      self._startRunCallbacks(fail)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks      self._runCallbacks()  --- <exception caught here> ---    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks      current.result = callback(current.result, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1171, in checkKnownErrors      key = error.trap(*command.allErrors)    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap      self.raiseException()    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException      raise self.value.with_traceback(self.tb)    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext      result = inContext.theWork()    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>      inContext.theWork = lambda: context.call(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext      return self.currentContext().callWithContext(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext      return func(*args,**kw)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext      return func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper      result = func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 755, in call_within_transaction      with connected(), post_commit_hooks:    File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__      return next(self.gen)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 684, in connected      connection.ensure_connection()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__      six.reraise(dj_exc_type, dj_exc_value, traceback)    File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise      raise value.with_traceback(tb)    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 189, in connect      self.connection = self.get_new_connection(conn_params)    File "/usr/lib/python3/dist-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection      connection = Database.connect(**conn_params)    File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 130, in connect      conn = _connect(dsn, connection_factory=connection_factory, **kwasync)  django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections I had a MAAS that has 2 region/racks + 2 racks controllers. 2 physical machines, 1 deployed as a pod. All of the sudden I started seeing this issue. After i noticed this issue, I also noticed my secondary region/rack was dead and these logs are from the primary region/rack. Lastly, I manually restarted regiond on the primary region/rack and things resolved themselves. 2018-10-25 03:55:05 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'GetSyslogConfiguration' command. [maas00:pid=12227:cmd=GetSyslogConfiguration:ask=bb38]  Traceback (most recent call last):    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 267, in run      self._asyncioEventloop.run_forever()    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run      f(*args, **kwargs)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback      self._startRunCallbacks(fail)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks      self._runCallbacks()  --- <exception caught here> ---    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks      current.result = callback(current.result, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1171, in checkKnownErrors      key = error.trap(*command.allErrors)    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap      self.raiseException()    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException      raise self.value.with_traceback(self.tb)    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext      result = inContext.theWork()    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>      inContext.theWork = lambda: context.call(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext      return self.currentContext().callWithContext(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext      return func(*args,**kw)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext      return func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper      result = func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 755, in call_within_transaction      with connected(), post_commit_hooks:    File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__      return next(self.gen)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 684, in connected      connection.ensure_connection()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__      six.reraise(dj_exc_type, dj_exc_value, traceback)    File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise      raise value.with_traceback(tb)    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 189, in connect      self.connection = self.get_new_connection(conn_params)    File "/usr/lib/python3/dist-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection      connection = Database.connect(**conn_params)    File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 130, in connect      conn = _connect(dsn, connection_factory=connection_factory, **kwasync)  django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
2018-10-25 04:17:40 Andres Rodriguez description I had a MAAS that has 2 region/racks + 2 racks controllers. 2 physical machines, 1 deployed as a pod. All of the sudden I started seeing this issue. After i noticed this issue, I also noticed my secondary region/rack was dead and these logs are from the primary region/rack. Lastly, I manually restarted regiond on the primary region/rack and things resolved themselves. 2018-10-25 03:55:05 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'GetSyslogConfiguration' command. [maas00:pid=12227:cmd=GetSyslogConfiguration:ask=bb38]  Traceback (most recent call last):    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 267, in run      self._asyncioEventloop.run_forever()    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run      f(*args, **kwargs)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback      self._startRunCallbacks(fail)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks      self._runCallbacks()  --- <exception caught here> ---    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks      current.result = callback(current.result, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1171, in checkKnownErrors      key = error.trap(*command.allErrors)    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap      self.raiseException()    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException      raise self.value.with_traceback(self.tb)    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext      result = inContext.theWork()    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>      inContext.theWork = lambda: context.call(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext      return self.currentContext().callWithContext(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext      return func(*args,**kw)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext      return func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper      result = func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 755, in call_within_transaction      with connected(), post_commit_hooks:    File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__      return next(self.gen)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 684, in connected      connection.ensure_connection()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__      six.reraise(dj_exc_type, dj_exc_value, traceback)    File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise      raise value.with_traceback(tb)    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 189, in connect      self.connection = self.get_new_connection(conn_params)    File "/usr/lib/python3/dist-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection      connection = Database.connect(**conn_params)    File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 130, in connect      conn = _connect(dsn, connection_factory=connection_factory, **kwasync)  django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections I had a MAAS that has 2 region/racks + 2 racks controllers. 2 physical machines, 1 deployed as a pod. All of the sudden I started seeing this issue. After i noticed this issue, I also noticed my secondary region/rack was dead and these logs are from the primary region/rack. Lastly, I manually restarted regiond on the primary region/rack and things resolved themselves. Also, max_connections is set to 200. 2018-10-25 03:55:05 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'GetSyslogConfiguration' command. [maas00:pid=12227:cmd=GetSyslogConfiguration:ask=bb38]  Traceback (most recent call last):    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 267, in run      self._asyncioEventloop.run_forever()    File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run      f(*args, **kwargs)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback      self._startRunCallbacks(fail)    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks      self._runCallbacks()  --- <exception caught here> ---    File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks      current.result = callback(current.result, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1171, in checkKnownErrors      key = error.trap(*command.allErrors)    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap      self.raiseException()    File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException      raise self.value.with_traceback(self.tb)    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext      result = inContext.theWork()    File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>      inContext.theWork = lambda: context.call(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext      return self.currentContext().callWithContext(ctx, func, *args, **kw)    File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext      return func(*args,**kw)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext      return func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper      result = func(*args, **kwargs)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 755, in call_within_transaction      with connected(), post_commit_hooks:    File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__      return next(self.gen)    File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 684, in connected      connection.ensure_connection()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__      six.reraise(dj_exc_type, dj_exc_value, traceback)    File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise      raise value.with_traceback(tb)    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 213, in ensure_connection      self.connect()    File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 189, in connect      self.connection = self.get_new_connection(conn_params)    File "/usr/lib/python3/dist-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection      connection = Database.connect(**conn_params)    File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 130, in connect      conn = _connect(dsn, connection_factory=connection_factory, **kwasync)  django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
2018-10-25 14:43:05 Blake Rouse maas: status Triaged Incomplete
2018-10-29 14:54:26 Andres Rodriguez maas: milestone 2.5.0rc1 2.5.0
2018-12-21 11:24:46 Andres Rodriguez maas: milestone 2.5.0 2.5.x
2019-09-19 11:51:35 Adam Collard maas: status Incomplete Expired