snap post-refresh crashes on dbupgrade

[Bryan Larsen]

$ snap refresh --channel=2.8/beta maas
error: cannot perform the following tasks:
- Run post-refresh hook of "maas" snap if present (run hook "post-refresh":
-----
Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running migrations:
  Applying maasserver.0203_interface_node_name_duplicates_delete...Traceback (most recent call last):
  File "/snap/maas/6461/bin/maas-region", line 11, in <module>
    load_entry_point('maas==2.8.0b4', 'console_scripts', 'maas-region')()
  File "/snap/maas/6461/lib/python3.6/site-packages/maasserver/region_script.py", line 76, in run
    run_django(is_snap, is_devenv)
  File "/snap/maas/6461/lib/python3.6/site-packages/maasserver/region_script.py", line 65, in run_django
    management.execute_from_command_line()
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/snap/maas/6461/lib/python3.6/site-packages/maasserver/management/commands/dbupgrade.py", line 117, in handle
    call_command("migrate", interactive=False)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/snap/maas/6461/lib/python3.6/site-packages/maasserver/migrations/maasserver/0203_interface_node_name_duplicates_delete.py", line 30, in delete_duplicate_interfaces
    node = Node.objects.get(id=duplicate["node_id"])
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/snap/maas/6461/usr/lib/python3/dist-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
__fake__.DoesNotExist: Node matching query does not exist.
Traceback (most recent call last):
  File "/snap/maas/6461/bin/maas", line 11, in <module>
    load_entry_point('maas==2.8.0b4', 'console_scripts', 'maas')()
  File "/snap/maas/6461/lib/python3.6/site-packages/maascli/__init__.py", line 44, in main
    options.execute(options)
  File "/snap/maas/6461/lib/python3.6/site-packages/maascli/snappy.py", line 607, in __call__
    raise exc
  File "/snap/maas/6461/lib/python3.6/site-packages/maascli/snappy.py", line 604, in __call__
    self.handle(options)
  File "/snap/maas/6461/lib/python3.6/site-packages/maascli/snappy.py", line 1096, in handle
    sys.exit(migrate_db())
  File "/snap/maas/6461/lib/python3.6/site-packages/maascli/snappy.py", line 508, in migrate_db
    "dbupgrade",
  File "/snap/maas/6461/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/snap/maas/6461/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1.
-----)

[Alberto Donato (ack)]

Could you please run the following query on the maas database and paste the result:

select name, array_agg(node_id) from maasserver_interface group by name, node_id having count(node_id) > 1

[Alberto Donato (ack)]

Also, please

select id, name, type, acquired from maasserver_interface where node_id is null

[Bryan Larsen (bryan-larsen)]

How do I connect to the postgres database inside the snap?

# sudo -u snap_daemon psql -h /var/snap/maas/common/postgres/sockets
psql: error: could not connect to server: FATAL: role "snap_daemon" does not exist

# sudo -u postgres psql -h /var/snap/maas/common/postgres/sockets
could not change directory to "/var/snap/maas/common/postgres/sockets": Permission denied
psql: error: could not connect to server: could not connect to server: Permission denied
 Is the server running locally and accepting
 connections on Unix domain socket "/var/snap/maas/common/postgres/sockets/.s.PGSQL.5432"?

[Alberto Donato (ack)]

"sudo psql -U maas -h /var/snap/maas/common/postgres/sockets -d maasdb" should do it

[Bryan Larsen (bryan-larsen)]

maasdb=> select name, array_agg(node_id) from maasserver_interface group by name, node_id having count(node_id) > 1
maasdb-> ;
 name | array_agg
------+-----------
(0 rows)

maasdb=> select id, name, type, acquired from maasserver_interface where node_id is null;
 id | name | type | acquired
----+------+---------+----------
  5 | eth0 | unknown | f
 41 | eth0 | unknown | f
 42 | eth0 | unknown | f
 43 | eth0 | unknown | f
 44 | eth0 | unknown | f
 45 | eth0 | unknown | f
 46 | eth0 | unknown | f
 47 | eth0 | unknown | f
 48 | eth0 | unknown | f
 28 | eth0 | unknown | f
 29 | eth0 | unknown | f
 30 | eth0 | unknown | f
 31 | eth0 | unknown | f
 32 | eth0 | unknown | f
 33 | eth0 | unknown | f
 34 | eth0 | unknown | f
 35 | eth0 | unknown | f
 36 | eth0 | unknown | f
 37 | eth0 | unknown | f
 38 | eth0 | unknown | f
 39 | eth0 | unknown | f
 40 | eth0 | unknown | f
(22 rows)

Also, poking around I finally found the logs, which should be useful:

2020-05-28 14:59:14.045 UTC [104318] ERROR: column "dismissable" of relation "maasserver_notification" does not exist at character 135
2020-05-28 14:59:14.045 UTC [104318] STATEMENT: INSERT INTO "maasserver_notification" ("created", "updated", "ident", "user_id", "users", "admins", "message", "context", "category", "dismissable") VALUES ('2020-05-28T14:59:14.045363'::timestamp, '2020-05-28T14:59:14.045363'::timestamp, 'deprecation_MD1_users', NULL, true, false, 'The setup for this MAAS is deprecated and not suitable for production environments, as the database is running inside the snap.<br>Please contact your MAAS administrator.<br><a class=''p-link--external'' href=''https://maas.io/deprecations/MD1''>How to migrate the database out of the snap...</a>', '{}', 'warning', false) RETURNING "maasserver_notification"."id"

[Alberto Donato (ack)]

Oh, the entries from the second query are strange.

Any chance you can recognize what those interfaces are for?
You can repeat the second query and include the "mac_address" field to include that.

[Bryan Larsen (bryan-larsen)]

Some or all of these are unmanaged machines on my network. MAAS is doing DHCP for the whole network so I don't have two fighting DHCP servers.

[Alberto Donato (ack)]

Thanks, that's what we suspected as well.