2016-03-25 10:52:55 [RegionServer,6,127.0.0.1] Rack controller 'None' disconnected.

Bug #1561816 reported by Andres Rodriguez
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse

Bug Description

After upgrade to latest trunk

==> /var/log/maas/rackd.log <==
2016-03-25 10:52:54+0800 [ClusterClient,client] Rack controller REJECTED by the region (via trusty-maas9:pid=10784).
2016-03-25 10:52:54+0800 [ClusterClient,client] ClusterClient connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 54509) PEER:IPv4Address(TCP, '127.0.0.1', 5250))

==> /var/log/maas/regiond.log <==
2016-03-25 10:52:54 [RegionServer,11,127.0.0.1] Rack controller 'None' disconnected.
2016-03-25 10:52:54 [RegionServer,11,127.0.0.1] RegionServer connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 5250) PEER:IPv4Address(TCP, '127.0.0.1', 54509))
2016-03-25 10:52:55 [-] Failed to register rack controller 'None' into the database. Connection has been dropped.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line 570, in register
            log.err(exc, msg)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line 522, in register
            transactional(rack_controller.update_interfaces), interfaces)
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 516, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 351, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.5/contextlib.py", line 30, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 2987, in update_interfaces
            interface = self._update_interface(name, interfaces[name])
          File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 3015, in _update_interface
            return self._update_physical_interface(name, config)
          File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 3068, in _update_physical_interface
            interface.save()
          File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 29, in save
            return super(CleanSave, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/models/timestampedmodel.py", line 64, in save
            return super(TimestampedModel, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 734, in save
            force_update=force_update, update_fields=update_fields)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 762, in save_base
            updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 827, in _save_table
            forced_update)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 877, in _do_update
            return filtered._update(values) > 0
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 580, in _update
            return query.get_compiler(self.db).execute_sql(CURSOR)
          File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1062, in execute_sql
            cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
          File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
            cursor.execute(sql, params)
          File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
          File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
            six.reraise(dj_exc_type, dj_exc_value, traceback)
          File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
            raise value.with_traceback(tb)
          File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
        django.db.utils.OperationalError: deadlock detected
        DETAIL: Process 11352 waits for ExclusiveLock on tuple (1,65) of relation 32900 of database 32496; blocked by process 10932.
        Process 10932 waits for ShareLock on transaction 347393; blocked by process 11483.
        Process 11483 waits for ExclusiveLock on tuple (2,21) of relation 32900 of database 32496; blocked by process 11330.
        Process 11330 waits for ShareLock on transaction 347298; blocked by process 11420.
        Process 11420 waits for ShareLock on transaction 347375; blocked by process 11373.
        Process 11373 waits for ExclusiveLock on tuple (1,65) of relation 32900 of database 32496; blocked by process 11352.
        HINT: See server log for query details.

==> /var/log/maas/rackd.log <==
2016-03-25 10:52:55+0800 [ClusterClient,client] Rack controller REJECTED by the region (via trusty-maas9:pid=10784).
2016-03-25 10:52:55+0800 [ClusterClient,client] ClusterClient connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 54462) PEER:IPv4Address(TCP, '127.0.0.1', 5250))

==> /var/log/maas/regiond.log <==
2016-03-25 10:52:55 [RegionServer,6,127.0.0.1] Rack controller 'None' disconnected.
2016-03-25 10:52:55 [RegionServer,6,127.0.0.1] RegionServer connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 5250) PEER:IPv4Address(TCP, '127.0.0.1', 54462))

Related branches

Changed in maas:
milestone: none → 2.0.0
importance: Undecided → Critical
description: updated
summary: - test
+ 2016-03-25 10:52:55+0800 [ClusterClient,client] Rack controller REJECTED
+ by the region (via trusty-maas9:pid=10784).
summary: - 2016-03-25 10:52:55+0800 [ClusterClient,client] Rack controller REJECTED
- by the region (via trusty-maas9:pid=10784).
+ 2016-03-25 10:52:55 [RegionServer,6,127.0.0.1] Rack controller 'None'
+ disconnected.
Changed in maas:
status: New → In Progress
assignee: nobody → Blake Rouse (blake-rouse)
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.