[2.0a1] Failed to update this region's process and endpoints; unleashed:pid=28940 record's may be out of date

[Andres Rodriguez]

2016-02-25 09:59:12 [-] Failed to update this region's process and endpoints; unleashed:pid=28940 record's may be out of date
        Traceback (most recent call last):
          File "/usr/lib/python3.5/threading.py", line 862, in run
            self._target(*self._args, **self._kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 794, in worker
            return target()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 197, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/__init__.py", line 165, in call_with_lock
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 448, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 275, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.5/contextlib.py", line 30, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line 1018, in update
            process = RegionControllerProcess.objects.get(id=self.processId.value)
          File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 334, in get
            self.model._meta.object_name
        maasserver.models.regioncontrollerprocess.DoesNotExist: RegionControllerProcess matching query does not exist.

[Gavin Panella (allenap)]

Note to anyone fixing this:

  unleashed:pid=28940 record's may be out of date

could be drive-by corrected to:

  unleashed:pid=28940's records may be out of date

[Andres Rodriguez (andreserl)]

this is apparently preventing me from enlisting multiple nodes at the same time.

[Rob Sanderson (robsanderson)]

I have this issue to in: 2.0.0~alpha1+bzr4736-0ubuntu1

---
2016-03-12 10:23:54 [-] Failed to update this region's process and endpoints; maas1:pid=218 record's may be out of date
 Traceback (most recent call last):
   File "/usr/lib/python3.5/threading.py", line 862, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 794, in worker
     return target()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
     task()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
     task()
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
     result = inContext.theWork()
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 197, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/__init__.py", line 165, in call_with_lock
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 448, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 275, in retrier
     return func(*args, **kwargs)
   File "/usr/lib/python3.5/contextlib.py", line 30, in inner
     return func(*args, **kwds)
   File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line 1032, in update
     process = RegionControllerProcess.objects.get(id=self.processId.value)
   File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 334, in get
     self.model._meta.object_name
 maasserver.models.regioncontrollerprocess.DoesNotExist: RegionControllerProcess matching query does not exist.

[Blake Rouse (blake-rouse)]

With latest trunk I cannot reproduce this error. Might have been related to the deadlock issue and now that triggers are not ran at startup I can't reproduce this bug.

Going to mark as Fix Committed for now, if it can be reproduced please let me know. If this happens in the NEC lab, please leave it in that state and let me take a look.