Failed to upgrade 3.4.1 to 3.4.2

Bug #2065312 reported by maasuser1

This bug report will be marked for expiration in 44 days if no further activity occurs. (find out why)

6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Incomplete
Undecided
Unassigned

Bug Description

```
clouduser@maas:~$ sudo snap refresh maas --channel=3.4/stable
[sudo] password for clouduser:
error: cannot perform the following tasks:
- Run post-refresh hook of "maas" snap if present (run hook "post-refresh":
-----
Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running migrations:
  Applying maasserver.0318_add_port_to_forward_dns_servers...Traceback (most recent call last):
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.DuplicateColumn: column "port" of relation "maasserver_forwarddnsserver" already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/snap/maas/35359/bin/maas-region", line 8, in <module>
    sys.exit(run())
  File "/snap/maas/35359/lib/python3.10/site-packages/maasserver/region_script.py", line 81, in run
    run_django(is_snap, is_devenv)
  File "/snap/maas/35359/lib/python3.10/site-packages/maasserver/region_script.py", line 70, in run_django
    management.execute_from_command_line()
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/snap/maas/35359/lib/python3.10/site-packages/maasserver/management/commands/dbupgrade.py", line 125, in handle
    call_command(
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/migrations/operations/fields.py", line 104, in database_forwards
    schema_editor.add_field(
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 522, in add_field
    self.execute(sql, params)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/snap/maas/35359/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column "port" of relation "maasserver_forwarddnsserver" already exists

Traceback (most recent call last):
  File "/snap/maas/35359/bin/maas", line 8, in <module>
    sys.exit(main())
  File "/snap/maas/35359/lib/python3.10/site-packages/maascli/__init__.py", line 39, in main
    options.execute(options)
  File "/snap/maas/35359/lib/python3.10/site-packages/maascli/snap.py", line 481, in __call__
    raise exc
  File "/snap/maas/35359/lib/python3.10/site-packages/maascli/snap.py", line 478, in __call__
    self.handle(options)
  File "/snap/maas/35359/lib/python3.10/site-packages/maascli/snap.py", line 1037, in handle
    sys.exit(migrate_db())
  File "/snap/maas/35359/lib/python3.10/site-packages/maascli/snap.py", line 386, in migrate_db
    subprocess.check_call(
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/snap/maas/35359/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1.
-----)
```

Revision history for this message
maasuser1 (maasuser1) wrote :

Fixed by droped `port` column from `maasserver_forwarddnsserver` table.

https://bugs.launchpad.net/maas/+bug/2058898

Revision history for this message
Anton Troyanov (troyanov) wrote :

Hi maasuser1,

Is it possible that you were using 3.4/edge to install MAAS 3.4.1?
That would explain why you already had `port` column.

https://git.launchpad.net/maas/commit/?id=99b3d1f1fb61dbdc25000c3394faa547df5749bb
https://git.launchpad.net/maas/commit/?id=a552d252204f0abaaa84eee4d36024fa2498c61e
https://git.launchpad.net/maas/commit/?id=a5e22161ba48e57b5465e7b491de0e376655b8f2

We didn't notice any migration issues when upgrading our lab from 3.4.1 to 3.4.2 (while being on 3.4/stable)

Changed in maas:
status: New → Incomplete
Revision history for this message
maasuser1 (maasuser1) wrote :

Thank you, Anton, yes. I remember I installed 3.4/edge and then refreshed to 3.4.1/stable a few months ago.

Revision history for this message
Anton Troyanov (troyanov) wrote :

We're excited for community users to try the latest and greatest version of MAAS, but unfortunately we can't guarantee that edge [0] channel will always work.

I am glad that you were able to solve the issue.

May I ask you to share result of the following query? Just to ensure that manual column removal won't lead to other issues in the future:
```
select * from django_migrations
where app = 'maasserver'
order by applied desc limit 20;
```

[0]: Edge releases often include a moving stream of changes without QA or review promises and are typically built automatically by a CI process from an arbitrary source code snapshot. Often the CI will only publish after some sort of automatic QA passed, and code reviews remain a good practice, but these are project specific. Assume edge releases may break often.

Revision history for this message
maasuser1 (maasuser1) wrote :

Thanks for the information!

```csv
361,maasserver,0318_add_port_to_forward_dns_servers,2024-05-09 14:41:42.106137 +00:00
360,maasserver,0305_add_port_to_forward_dns_servers,2024-03-12 23:24:59.398346 +00:00
358,maasserver,0304_interface_params_no_autoconf,2023-12-21 19:30:53.587876 +00:00
357,maasserver,0303_interface_params_cleanups,2023-12-21 19:30:53.564463 +00:00
356,maasserver,0302_big_auto_field,2023-12-21 19:30:53.224573 +00:00
355,maasserver,0301_discovery_ignore_fks,2023-12-21 19:29:38.975402 +00:00
354,maasserver,0300_current_script_set_foreign_keys_readd,2023-12-21 19:29:38.445091 +00:00
353,maasserver,0299_current_script_set_foreign_keys_cleanup,2023-12-21 19:29:38.180414 +00:00
352,maasserver,0298_current_script_set_foreign_keys_drop_indexes,2023-12-21 19:29:37.995746 +00:00
351,maasserver,0297_move_metadata_script_models,2023-12-21 19:29:37.759516 +00:00
350,maasserver,0296_move_metadata_node_models,2023-12-21 19:29:37.300648 +00:00
347,maasserver,0295_macaddress_text_field,2023-12-21 19:29:36.533792 +00:00
346,maasserver,0294_keyring_data_binary_field,2023-12-21 19:29:36.056234 +00:00
345,maasserver,0293_drop_verbose_regex_validator,2023-12-21 19:29:36.032877 +00:00
344,maasserver,0292_use_builtin_json_field,2023-12-21 19:29:35.949926 +00:00
343,maasserver,0291_rdns_hostnames_as_array,2023-12-21 19:29:34.823411 +00:00
334,maasserver,0290_migrate_node_power_parameters,2023-12-21 14:04:55.230509 +00:00
333,maasserver,0289_vault_secret,2023-12-21 14:04:55.119993 +00:00
332,maasserver,0288_rootkey_material_secret,2023-12-21 14:04:55.025346 +00:00
331,maasserver,0287_add_controller_info_vault_flag,2023-12-21 14:04:54.950965 +00:00

```

Revision history for this message
Anton Troyanov (troyanov) wrote :

Oh, this doesn't seem to be right and might lead to other issues in the future :(

```
361,maasserver,0318_add_port_to_forward_dns_servers,2024-05-09 14:41:42.106137 +00:00
360,maasserver,0305_add_port_to_forward_dns_servers,2024-03-12 23:24:59.398346 +00:00
```

So I guess one came from [0] and another one from [1]

[0]: https://git.launchpad.net/maas/commit/?id=99b3d1f1fb61dbdc25000c3394faa547df5749bb
[1]: https://git.launchpad.net/maas/commit/?id=a5e22161ba48e57b5465e7b491de0e376655b8f2

Let's do an open surgery on the database? Please create a backup before proceeding!

```sql
# drop duplicate migrations
delete from django_migrations where id in (360, 361);
# drop port column
alter table maasserver_forwarddnsserver drop column port;
```

Once the database is cleaned up, please run `maas migrate` which should apply only one `add_port_to_forward_dns_servers` migration.

You should end up with something like this:
```csv
{new ID},maasserver,0318_add_port_to_forward_dns_servers, {timestamp}
358,maasserver,0304_interface_params_no_autoconf,2023-12-21 19:30:53.587876 +00:00
```

Changed in maas:
status: Incomplete → New
status: New → Incomplete
Revision history for this message
maasuser1 (maasuser1) wrote :

Thanks Anton.

```
362,maasserver,0318_add_port_to_forward_dns_servers,2024-05-11 09:21:31.230599 +00:00
358,maasserver,0304_interface_params_no_autoconf,2023-12-21 19:30:53.587876 +00:00
357,maasserver,0303_interface_params_cleanups,2023-12-21 19:30:53.564463 +00:00
356,maasserver,0302_big_auto_field,2023-12-21 19:30:53.224573 +00:00
355,maasserver,0301_discovery_ignore_fks,2023-12-21 19:29:38.975402 +00:00
```

Changed in maas:
status: Incomplete → New
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.