Activity log for bug #1401983

Date Who What changed Old value New value Message
2014-12-12 16:59:32 Jason Hobbs bug added bug
2014-12-12 17:03:19 Jason Hobbs description I have a MAAS with 100+ nodes in it, and in its maas-django.log I'm seeing these errors from time to time, especially if i try to start many nodes at once: ERROR 2014-12-12 16:30:25,747 maasserver ################################ Exception: deadlock detected DETAIL: Process 746 waits for ShareLock on transaction 43344353; blocked by process 742. Process 742 waits for ExclusiveLock on tuple (5,3) of relation 16585 of database 16385; blocked by process 746. HINT: See server log for query details. ################################ ERROR 2014-12-12 16:30:25,747 maasserver Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python2.7/dist-packages/maasserver/api/pxeconfig.py", line 171, in pxeconfig node.save() File "/usr/lib/python2.7/dist-packages/maasserver/models/cleansave.py", line 38, in save return super(CleanSave, self).save(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/maasserver/models/timestampedmodel.py", line 55, in save return super(TimestampedModel, self).save(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 545, in save force_update=force_update, update_fields=update_fields) File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 573, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 635, in _save_table forced_update) File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 679, in _do_update return filtered._update(values) > 0 File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 507, in _update return query.get_compiler(self.db).execute_sql(None) File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 975, in execute_sql cursor = super(SQLUpdateCompiler, self).execute_sql(result_type) File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql cursor.execute(sql, params) File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) OperationalError: deadlock detected DETAIL: Process 746 waits for ShareLock on transaction 43344353; blocked by process 742. Process 742 waits for ExclusiveLock on tuple (5,3) of relation 16585 of database 16385; blocked by process 746. HINT: See server log for query details. I have a MAAS with 100+ nodes in it, and in its maas-django.log I'm seeing these errors from time to time, especially if I try to start many nodes at once. ERROR 2014-12-12 16:30:25,747 maasserver ################################ Exception: deadlock detected DETAIL: Process 746 waits for ShareLock on transaction 43344353; blocked by process 742. Process 742 waits for ExclusiveLock on tuple (5,3) of relation 16585 of database 16385; blocked by process 746. HINT: See server log for query details.  ################################ ERROR 2014-12-12 16:30:25,747 maasserver Traceback (most recent call last):   File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response     response = wrapped_callback(request, *callback_args, **callback_kwargs)   File "/usr/lib/python2.7/dist-packages/maasserver/api/pxeconfig.py", line 171, in pxeconfig     node.save()   File "/usr/lib/python2.7/dist-packages/maasserver/models/cleansave.py", line 38, in save     return super(CleanSave, self).save(*args, **kwargs)   File "/usr/lib/python2.7/dist-packages/maasserver/models/timestampedmodel.py", line 55, in save     return super(TimestampedModel, self).save(*args, **kwargs)   File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 545, in save     force_update=force_update, update_fields=update_fields)   File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 573, in save_base     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)   File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 635, in _save_table     forced_update)   File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 679, in _do_update     return filtered._update(values) > 0   File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 507, in _update     return query.get_compiler(self.db).execute_sql(None)   File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 975, in execute_sql     cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)   File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql     cursor.execute(sql, params)   File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute     return self.cursor.execute(sql, params)   File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__     six.reraise(dj_exc_type, dj_exc_value, traceback)   File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute     return self.cursor.execute(sql, params) OperationalError: deadlock detected DETAIL: Process 746 waits for ShareLock on transaction 43344353; blocked by process 742. Process 742 waits for ExclusiveLock on tuple (5,3) of relation 16585 of database 16385; blocked by process 746. HINT: See server log for query details.
2014-12-12 19:21:46 Launchpad Janitor branch linked lp:~blake-rouse/maas/fix-1401983
2014-12-12 19:24:01 Blake Rouse maas: status New In Progress
2014-12-12 19:24:02 Blake Rouse maas: importance Undecided Critical
2014-12-12 19:24:05 Blake Rouse maas: assignee Blake Rouse (blake-rouse)
2014-12-12 19:24:09 Blake Rouse nominated for series maas/1.7
2014-12-12 19:24:09 Blake Rouse bug task added maas/1.7
2014-12-12 19:24:14 Blake Rouse maas/1.7: milestone 1.7.2
2014-12-12 19:24:15 Blake Rouse maas/1.7: assignee Blake Rouse (blake-rouse)
2014-12-12 19:24:17 Blake Rouse maas/1.7: importance Undecided Critical
2014-12-12 19:24:19 Blake Rouse maas/1.7: status New In Progress
2014-12-12 21:39:22 MAAS Lander maas: status In Progress Fix Committed
2014-12-12 21:54:33 Launchpad Janitor branch linked lp:~blake-rouse/maas/fix-1401983-1.7
2014-12-12 22:12:48 MAAS Lander maas/1.7: status In Progress Fix Committed
2014-12-16 14:43:47 Christian Reis maas/1.7: milestone 1.7.2 1.7.1
2015-02-05 07:00:10 Launchpad Janitor branch linked lp:ubuntu/vivid-proposed/maas
2015-02-05 09:28:10 Andres Rodriguez maas/1.7: status Fix Committed Fix Released
2015-02-05 09:28:12 Andres Rodriguez maas: status Fix Committed Fix Released
2015-07-28 07:15:31 Andres Rodriguez tags verification-done