Failure creating KVM pod in CI

Bug #1845459 reported by Björn Tillenius
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Björn Tillenius
2.6
Fix Released
High
Björn Tillenius

Bug Description

The test_machines_deployed test fails in CI now and then. Some nodes get in a Failed Deployment state with an error like this:

  Internal error while creating KVM pod. (See regiond.log for details.)

The regiond.log contains this traceback:

2019-09-26 07:41:42 regiond: [info] 10.245.136.6 POST /MAAS/metadata/status/hkby
cw HTTP/1.0 --> 204 NO_CONTENT (referrer: -; agent: python-requests/2.18.4)
2019-09-26 07:41:42 maasserver.models.node: [debug] Unable to format event {'hos
tname': 'epic-cat', 'config': <function Node.mark_failed.<locals>.<lambda> at 0x
7effb8932ae8>, 'log_logger': <LegacyLogger 'maasserver.models.node'>, 'log_level
': <LogLevel=debug>, 'log_namespace': 'maasserver.models.node', 'log_source': No
ne, 'log_format': "Node '{hostname}' failed deployment with curtin config: {conf
ig()}", 'log_time': 1569483702.8679438}: get_curtin_merged_config() missing 1 re
quired positional argument: 'node'
2019-09-26 07:41:42 metadataserver.api_twisted: [critical] Exception while savin
g pod form.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 75
6, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 56
3, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.6/contextlib.py", line 52, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/metadataserver/api_twisted.py", l
ine 390, in _processMessage
            _create_pod_for_deployment(node)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/metadataserver/api_twisted.py", l
ine 213, in _create_pod_for_deployment
            pod_form.save()
          File "/usr/lib/python3/dist-packages/maasserver/forms/pods.py", line 2
81, in save
            self.instance = update_obj(self.instance)
          File "/usr/lib/python3/dist-packages/maasserver/forms/pods.py", line 2
53, in update_obj
            kernel_opts="console=tty1 console=ttyS0")
          File "/usr/lib/python3/dist-packages/django/db/models/manager.py", lin
e 85, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
466, in get_or_create
            return self._create_object_from_params(lookup, params)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
506, in _create_object_from_params
            six.reraise(*exc_info)
          File "/usr/lib/python3/dist-packages/django/utils/six.py", line 686, i
n reraise
            raise value
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 498, in _create_object_from_params
            obj = self.create(**params)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 394, in create
            obj.save(force_insert=True, using=self.db)
          File "/usr/lib/python3/dist-packages/maasserver/models/tag.py", line 151, in save
            super(Tag, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 185, in save
            return super(CleanSave, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/models/timestampedmodel.py", line 90, in save
            return super(TimestampedModel, self).save(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 808, in save
            force_update=force_update, update_fields=update_fields)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 838, in save_base
            updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
          File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 249, in _save_table
            update_fields=update_fields)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 924, in _save_table
            result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
          File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 963, in _do_insert
            using=using, raw=raw)
          File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 1076, in _insert
            return query.get_compiler(using=using).execute_sql(return_id)
          File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql
            cursor.execute(sql, params)
          File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
          File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__
            six.reraise(dj_exc_type, dj_exc_value, traceback)
          File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
            raise value.with_traceback(tb)
          File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
        django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_tag_name_key"
        DETAIL: Key (name)=(pod-console-logging) already exists.

Related branches

Changed in maas:
status: New → Triaged
importance: Undecided → High
milestone: none → 2.7.0alpha1
Changed in maas:
status: Triaged → In Progress
assignee: nobody → Björn Tillenius (bjornt)
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.