Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist

[Xav Paice]

Upgrading an older Trusty based Maas (1.9.5) to Xenial (2.3.5) resulted in a traceback during the do-release-upgrade procedure:

  Applying maasserver.0011_domain_data...Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: column maasserver_domain.ttl does not exist
LINE 1: ...ain"."name", "maasserver_domain"."authoritative", "maasserve...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/maasserver/__main__.py", line 9, in <module>
    raise SystemExit(execute_from_command_line())
  File "/usr/lib/python3/dist-packages/maasserver/__init__.py", line 73, in execute_from_command_line
    management.execute_from_command_line()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/maasserver/management/commands/dbupgrade.py", line 426, in handle
    fake_initial=self._south_was_performed(database))
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3/dist-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/lib/python3/dist-packages/maasserver/migrations/builtin/maasserver/0011_domain_data.py", line 109, in migrate_staticipaddress_hostname
    domain_id = maasserver.models.dnsresource.get_default_domain()
  File "/usr/lib/python3/dist-packages/maasserver/models/dnsresource.py", line 57, in get_default_domain
    return Domain.objects.get_default_domain().id
  File "/usr/lib/python3/dist-packages/maasserver/models/domain.py", line 108, in get_default_domain
    'updated': now,
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 405, in get_or_create
    return self.get(**lookup), False
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column maasserver_domain.ttl does not exist
LINE 1: ...ain"."name", "maasserver_domain"."authoritative", "maasserve...
                                                             ^

dpkg: error processing package maas-region-controller (--configure):
 subprocess installed post-installation script returned error exit status 1
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          dpkg: dependency problems prevent configuration of maas:
 maas depends on maas-region-controller (= 2.3.5-6511-gf466fdb-0ubuntu1); however:
  Package maas-region-controller is not configured yet.

To reproduce, install Maas on Trusty, with a working database, and run do-release-upgrade.

I suspect there's some edge case in this particular database, working now on trying to identify it. The particular database has been around for at least 3 years now.

[Xav Paice (xavpaice)]

Given this is a test, I tried adding the missing column:

postgres=# \connect maasdb
You are now connected to database "maasdb" as user "postgres".
maasdb=# \d maasserver_domain
                                     Table "public.maasserver_domain"
    Column | Type | Modifiers
---------------+--------------------------+----------------------------------------------------------------
 id | integer | not null default nextval('maasserver_domain_id_seq'::regclass)
 created | timestamp with time zone | not null
 updated | timestamp with time zone | not null
 name | character varying(256) | not null
 authoritative | boolean |
Indexes:
    "maasserver_domain_pkey" PRIMARY KEY, btree (id)
    "maasserver_domain_name_key" UNIQUE CONSTRAINT, btree (name)
    "maasserver_domain_946f3fba" btree (authoritative)
    "maasserver_domain_name_4403c8e5d405ce97_like" btree (name varchar_pattern_ops)
Referenced by:
    TABLE "maasserver_dnsresource" CONSTRAINT "maasserver_d_domain_id_48f70cfc2a2704fa_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "maasserver_node" CONSTRAINT "maasserver_n_domain_id_67d5fcf688c8fcd5_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED

maasdb=# alter table maasserver_domain add column ttl integer;
ALTER TABLE
maasdb=# \d maasserver_domain
                                     Table "public.maasserver_domain"
    Column | Type | Modifiers
---------------+--------------------------+----------------------------------------------------------------
 id | integer | not null default nextval('maasserver_domain_id_seq'::regclass)
 created | timestamp with time zone | not null
 updated | timestamp with time zone | not null
 name | character varying(256) | not null
 authoritative | boolean |
 ttl | integer |
Indexes:
    "maasserver_domain_pkey" PRIMARY KEY, btree (id)
    "maasserver_domain_name_key" UNIQUE CONSTRAINT...

[Felipe Reyes (freyes)]

I could reproduce this issue locally, and this is what I believe is happening with this environment (and not for other upgrades) is that the database has no domains defined for the interfaces related, so this piece of code [0] is falling into the "else" section which calls get_default_domain() method and internally ends up query the 'ttl' column. So the migration step 0011 is relying on it and expecting that it was created in step 0010 and it wasn't, it was added way later in migration step 0023.

[0] https://github.com/maas/maas/blob/2.3/src/maasserver/migrations/builtin/maasserver/0011_domain_data.py#L106-L109

[Dougal Matthews (d0ugal)]

Yup, as Felipe links I think the problem here is that we are using the models within the database migration. However, the models assume all migrations have been applied. The call to `maasserver.models.dnsresource.get_default_domain()` should be replaced with something that doesn't use the models.

[Jerzy Husakowski (jhusakowski)]

MAAS upgrade logic should not rely on MAAS models sources. This issue appeared on an old version of MAAS, but may appear in other upgrade scenarios. We will fix this on the current MAAS versions and do not plan to backport the fix to older versions.

[Jerzy Husakowski (jhusakowski)]

This issue requires further analysis of MAAS upgrade mechanism, to determine if it's possible to prevent such problems, e.g. by eliminating references to MAAS code in migration logic