We upgraded MAAS 1.8.3+bzr4053-0ubuntu1~trusty1 to 1.9.0+bzr4533-0ubuntu1~trusty1 and started getting these errors in the logs:
maas.log:
Jan 21 11:22:36 atlas maas.lease_upload_service: [ERROR] Failed to upload leases: ('UNHANDLED', 'Unknown Error [atlas:pid=23547:cmd=UpdateLeases:ask=93f]')
regiond.log:
2016-01-21 11:25:36 [-] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the UpdateLeases command. [atlas:pid=23547:cmd=UpdateLeases:ask=9
4b]
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 791, in _worker
return super(ThreadPool, self)._worker()
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 200, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 501, in call_within_transaction
return func_outside_txn(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 328, in retrier
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/maasserver/rpc/leases.py", line 51, in update_leases
StaticIPAddress.objects.update_leases(nodegroup, leases)
File "/usr/lib/python2.7/dist-packages/maasserver/models/staticipaddress.py", line 506, in update_leases
subnet=subnet)
File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 157, in create
return self.get_queryset().create(**kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 319, in create
obj.save(force_insert=True, using=self.db)
File "/usr/lib/python2.7/dist-packages/maasserver/models/cleansave.py", line 38, in save
return super(CleanSave, self).save(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/maasserver/models/timestampedmodel.py", line 70, in save
return super(TimestampedModel, self).save(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 545, in save
force_update=force_update, update_fields=update_fields)
File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 573, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 654, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 687, in _do_insert
using=using, raw=raw)
File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 232, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 1511, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 898, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key"
DETAIL: Key (ip)=(10.96.4.45) already exists.
I'm almost sure not all nodes were recommissioned after the upgrade. I'll do that now.
Please let us know if recommissioning fixes things. Also, a copy of /var/lib/