ImproperlyConfigured django error

[Tycho Andersen]

from celery.log:

[2014-02-26 18:43:01,409: CRITICAL/MainProcess] Can't decode message body: ImproperlyConfigured('Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.',) (type:u'application/x-python-serialize' encoding:u'binary' raw:"'\\x80\\x02}q\\x01(U\\x07expiresq\\x02NU\\x03utcq\\x03\\x88U\\x04argsq\\x04]q\\x05X\\x03\\x00\\x00\\x00amtq\\x06aU\\x05chordq\\x07NU\\tcallbacksq\\x08NU\\x08errbacksq\\tNU\\x07tasksetq\\nNU\\x02idq\\x0bU$3f297762-60bb-448d-8694-ba594757d937q\\x0cU\\x07retriesq\\rK\\x00U\\x04taskq\\x0eU!provisioningserver.tasks.power_onq\\x0fU\\ttimelimitq\\x10NN\\x86U\\x03etaq\\x11NU\\x06kwargsq\\x12}q\\x13(X\\x08\\x00\\x00\\x00usernameq\\x14X\\x00\\x00\\x00\\x00X\\t\\x00\\x00\\x00ipmipowerq\\x15X\\x13\\x00\\x00\\x00/usr/sbin/ipmipowerq\\x16X\\x08\\x00\\x00\\x00ipmitoolq\\x17X\\x11\\x00\\x00\\x00/usr/bin/ipmitoolq\\x18X\\t\\x00\\x00\\x00fence_cduq\\x19X\\x13\\x00\\x00\\x00/usr/sbin/fence_cduq\\x1aX\\r\\x00\\x00\\x00power_addressq\\x1bX\\x17\\x00\\x00\\x00qemu://localhost/systemq\\x1cX\\x08\\x00\\x00\\x00power_idq\\x1dX)\\x00\\x00\\x00node-0e2f2410-9f05-11e3-86b2-00ee4c62088dq\\x1eX\\x13\\x00\\x00\\x00ipmi_chassis_configq\\x1fX\\x1d\\x00\\x00\\x00/usr/sbin/ipmi-chassis-configq... (789b)"')
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 583, in _receive_callback
    decoded = None if on_m else message.decode()
  File "/usr/lib/python2.7/dist-packages/kombu/message.py", line 123, in decode
    self.content_encoding, accept=self.accept)
  File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 165, in loads
    return decode(data)
  File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 49, in pickle_loads
    return load(BytesIO(s))
  File "/usr/lib/python2.7/dist-packages/maasserver/fields.py", line 31, in <module>
    from django.db.models import (
  File "/usr/lib/python2.7/dist-packages/django/db/models/__init__.py", line 5, in <module>
    from django.db.models.query import Q
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 17, in <module>
    from django.db.models.deletion import Collector
  File "/usr/lib/python2.7/dist-packages/django/db/models/deletion.py", line 4, in <module>
    from django.db.models import signals, sql
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/__init__.py", line 4, in <module>
    from django.db.models.sql.subqueries import *
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/subqueries.py", line 12, in <module>
    from django.db.models.sql.query import Query
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 22, in <module>
    from django.db.models.sql import aggregates as base_aggregates_module
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/aggregates.py", line 9, in <module>
    ordinal_aggregate_field = IntegerField()
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 116, in __init__
    self.db_tablespace = db_tablespace or settings.DEFAULT_INDEX_TABLESPACE
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 47, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

sudo service maas-cluster-celery restart seems to fix it.

[Tycho Andersen (tycho-s)]

This seems to happen after I restart the celery server manually via 'sudo service maas-cluster-celery restart'.

[Julian Edwards (julian-edwards)]

So are you saying that restarting it manually *sometimes* makes it produce this error?

[Tycho Andersen (tycho-s)]

On Thu, Feb 27, 2014 at 12:54:35AM -0000, Julian Edwards wrote:
> So are you saying that restarting it manually *sometimes* makes it
> produce this error?

Yeah, it seemed that way earlier, although now it seems like it
/always/ gets me this error.

[Julian Edwards (julian-edwards)]

I just tried and I don't get this, after many restarts.

[Julian Edwards (julian-edwards)]

Can you provide a bit more info about the package version, like it says in the bug reporting guidelines that nobody reads :)

[Tycho Andersen (tycho-s)]

On Thu, Feb 27, 2014 at 02:49:47AM -0000, Julian Edwards wrote:
> Can you provide a bit more info about the package version, like it says
> in the bug reporting guidelines that nobody reads :)

Yep, this is trusty maas:

CloudFive:/var/log/maas$ dpkg -l | grep maas
ii maas
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii maas-cli
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Client
Tool
ii maas-cluster-controller
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS
Cluster Controller
ii maas-common
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii maas-dhcp
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
- DHCP Configuration (meta-package)
ii maas-dns
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
- DNS configuration (metapackage)
ii maas-region-controller
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii maas-region-controller-min
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii python-django-maas
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
- (django files)
ii python-maas-client
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS API
Client - (python files)
ii python-maas-provisioningserver
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server

I'd provide more info here, but to be honest I don't know what would
be useful. I know very little about django. I'd debug if I could, but
it just seems as though I restart either apache2 or
maas-cluster-celery and I get this.

[Julian Edwards (julian-edwards)]

Ok I'll wait for Raphaël to comment as he's the Django expert. I can't
see why this would only happen for you though :/

Out of interest, you only have the packages installed right? You're not
in a directory that has the dev environment? They don't mix well!

[Tycho Andersen (tycho-s)]

On Thu, Feb 27, 2014 at 03:41:15AM -0000, Julian Edwards wrote:
> Ok I'll wait for Raphaël to comment as he's the Django expert. I can't
> see why this would only happen for you though :/
>
> Out of interest, you only have the packages installed right? You're not
> in a directory that has the dev environment? They don't mix well!

Yep, just the packages.

[Raphaël Badin (rvb)]

 File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 49, in pickle_loads
    return load(BytesIO(s))
  File "/usr/lib/python2.7/dist-packages/maasserver/fields.py", line 31, in <module>
    from django.db.models import (

^ as you can see, the de-serializer is trying to access maasserver's code and it shouldn't do that. The import partially succeeds because the cluster happens to be running on the same machine but the cluster celery doesn't have access to Django's settings, hence the error.

It means the "power up" task code was changed recently and it is either calling MAAS code directly or sending MAAS objects over the wire.

[Raphaël Badin (rvb)]

btw, Diogo just got the same error in the lab.

[Diogo Matsubara (matsubara)]

I can reliably reproduce this in the qa lab: http://d-jenkins.ubuntu-ci:8080/view/MAAS/job/trusty-adt-maas-manual/65/console where the test fails to juju bootstrap a node and I can see the same exception Tycho reported in the celery.log. In the CI tests we don't restart celery but we do restart maas-pserv

[Raphaël Badin (rvb)]

This needs investigating but I suspect this change https://code.launchpad.net/~julian-edwards/maas/ipmi-with-mac/+merge/208286 is responsible for the problem. Well, the problem was there before this change but the change made the problem worse in a way.
'test_start_nodes_wakeonlan_falls_back_to_primary_mac' proves that the object passed to the task is an object of type maasserver.fields.MAC. It should be a string.

[Julian Edwards (julian-edwards)]

!! Nicely sleuthed!

[Julian Edwards (julian-edwards)]

We need a test for this kind of cross-project pollution, it's far too easy to return objects that can't be unserialized in the clusters. Answers on a postcard ...