Failure creating KVM pod in CI

Bug #1845459 reported by Björn Tillenius on 2019-09-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
High
Björn Tillenius
2.6
High
Björn Tillenius

Bug Description

The test_machines_deployed test fails in CI now and then. Some nodes get in a Failed Deployment state with an error like this:

  Internal error while creating KVM pod. (See regiond.log for details.)

The regiond.log contains this traceback:

2019-09-26 07:41:42 regiond: [info] 10.245.136.6 POST /MAAS/metadata/status/hkby
cw HTTP/1.0 --> 204 NO_CONTENT (referrer: -; agent: python-requests/2.18.4)
2019-09-26 07:41:42 maasserver.models.node: [debug] Unable to format event {'hos
tname': 'epic-cat', 'config': <function Node.mark_failed.<locals>.<lambda> at 0x
7effb8932ae8>, 'log_logger': <LegacyLogger 'maasserver.models.node'>, 'log_level
': <LogLevel=debug>, 'log_namespace': 'maasserver.models.node', 'log_source': No
ne, 'log_format': "Node '{hostname}' failed deployment with curtin config: {conf
ig()}", 'log_time': 1569483702.8679438}: get_curtin_merged_config() missing 1 re
quired positional argument: 'node'
2019-09-26 07:41:42 metadataserver.api_twisted: [critical] Exception while savin
g pod form.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 75
6, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 56
3, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.6/contextlib.py", line 52, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/metadataserver/api_twisted.py", l
ine 390, in _processMessage
            _create_pod_for_deployment(node)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/metadataserver/api_twisted.py", l
ine 213, in _create_pod_for_deployment
            pod_form.save()
          File "/usr/lib/python3/dist-packages/maasserver/forms/pods.py", line 2
81, in save
            self.instance = update_obj(self.instance)
          File "/usr/lib/python3/dist-packages/maasserver/forms/pods.py", line 2
53, in update_obj
            kernel_opts="console=tty1 console=ttyS0")
          File "/usr/lib/python3/dist-packages/django/db/models/manager.py", lin
e 85, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
466, in get_or_create
            return self._create_object_from_params(lookup, params)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
506, in _create_object_from_params
            six.reraise(*exc_info)
          File "/usr/lib/python3/dist-packages/django/utils/six.py", line 686, i
n reraise
            raise value
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 498, in _create_object_from_params
            obj = self.create(**params)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 394, in create
            obj.save(force_insert=True, using=self.db)
          File "/usr/lib/python3/dist-packages/maasserver/models/tag.py", line 151, in save
            super(Tag, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 185, in save
            return super(CleanSave, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/models/timestampedmodel.py", line 90, in save
            return super(TimestampedModel, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 808, in save
            force_update=force_update, update_fields=update_fields)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 838, in save_base
            updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
          File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 249, in _save_table
            update_fields=update_fields)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 924, in _save_table
            result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 963, in _do_insert
            using=using, raw=raw)
          File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 1076, in _insert
            return query.get_compiler(using=using).execute_sql(return_id)
          File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql
            cursor.execute(sql, params)
          File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
          File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__
            six.reraise(dj_exc_type, dj_exc_value, traceback)
          File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
            raise value.with_traceback(tb)
          File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
        django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_tag_name_key"
        DETAIL: Key (name)=(pod-console-logging) already exists.

Related branches

Changed in maas:
status: New → Triaged
importance: Undecided → High
milestone: none → 2.7.0alpha1
Changed in maas:
status: Triaged → In Progress
assignee: nobody → Björn Tillenius (bjornt)
Changed in maas:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers