RackControllerService flooding log with: 'RegionAdvertisingService' object has no attribute 'processId'

Bug #1578800 reported by Lee Trager on 2016-05-05
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Critical
Gavin Panella

Bug Description

I pulled trunk today and I'm noticing that regiond.log is being flooded with exceptions from the RackControllerService. This happens in both the dev(make test) and a live environment.

==> logs/regiond/current <==
2016-05-05 19:51:54 [-] Log opened.
2016-05-05 19:51:54 [-] twistd 16.0.0 (/usr/bin/python3.5 3.5.1) starting up.
2016-05-05 19:51:54 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2016-05-05 19:51:54 [-] Site starting on 5240
2016-05-05 19:51:54 [-] Factory starting on 5250
2016-05-05 19:51:54 [-] Traceback (most recent call last):
2016-05-05 19:51:54 [-] File "/home/lee/maas/trunk/bin/twistd.region", line 40, in <module>
2016-05-05 19:51:54 [-] sys.exit(twisted.scripts.twistd.run())
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/twistd.py", line 29, in run
2016-05-05 19:51:54 [-] app.run(runApp, ServerOptions)
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 617, in run
2016-05-05 19:51:54 [-] runApp(config)
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/twistd.py", line 25, in runApp
2016-05-05 19:51:54 [-] _SomeApplicationRunner(config).run()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 352, in run
2016-05-05 19:51:54 [-] self.postApplication()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/_twistd_unix.py", line 207, in postApplication
2016-05-05 19:51:54 [-] self.startApplication(self.application)
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/_twistd_unix.py", line 403, in startApplication
2016-05-05 19:51:54 [-] app.startApplication(application, not self.config['no_save'])
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 633, in startApplication
2016-05-05 19:51:54 [-] service.IService(application).startService()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 283, in startService
2016-05-05 19:51:54 [-] service.startService()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 283, in startService
2016-05-05 19:51:54 [-] service.startService()
2016-05-05 19:51:54 [-] File "/home/lee/maas/trunk/src/provisioningserver/utils/twisted.py", line 140, in wrapper
2016-05-05 19:51:54 [-] return func(*args, **kwargs)
2016-05-05 19:51:54 [-] File "/home/lee/maas/trunk/src/maasserver/rack_controller.py", line 125, in startService
2016-05-05 19:51:54 [-] self.starting = self.advertisingService.processId.get()
2016-05-05 19:51:54 [-] AttributeError: 'RegionAdvertisingService' object has no attribute 'processId'
/home/lee/maas/trunk/src/maasserver/websockets/protocol.py:27: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
  from django.utils.importlib import import_module

Related branches

Lee Trager (ltrager) on 2016-05-05
summary: - advertisingService flooding log with exceptions
+ RackControllerService flooding log with exceptions

introduced in 4986: [r=mpontillo][bug=][author=allenap] Address situations where RegionAdvertisingService was not thread-safe.

Lee Trager (ltrager) wrote :

I think there may be a hidden conflict between 4985 and 4986. If I run 4985 I don't see the problem. However if I take trunk and revert 4986 the problem still exists, if I revert 4985 it goes away.

Gavin Panella (allenap) on 2016-05-06
Changed in maas:
status: New → Triaged
status: Triaged → In Progress
LaMont Jones (lamont) on 2016-05-06
summary: - RackControllerService flooding log with exceptions
+ RackControllerService flooding log with: 'RegionAdvertisingService'
+ object has no attribute 'processId'
Changed in maas:
status: In Progress → Fix Committed
Gordon Bennett (gor-blimey) wrote :

I tried this patch on 2.0.0~beta5+bzr4998-0ubuntu1~xenial1
and although the GUI does now open the regiond.log still reports errors related to RegionAdvertisingService.

Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 393, in callback
     self._startRunCallbacks(result)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
     self._runCallbacks()
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
     return _inlineCallbacks(None, gen, Deferred())
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
     result = g.send(result)
   File "/usr/lib/python3/dist-packages/maasserver/service_monitor.py", line 101, in _updateDatabase
     processId = yield self.advertisingService.processId.get()
 builtins.AttributeError: 'RegionAdvertisingService' object has no attribute 'processId'

Gavin Panella (allenap) wrote :

Gordon, I'm going to look into that tomorrow morning. Thanks for letting us know.

Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers