Migration during the upgrade to 3.4 stable is failing for MAAS instances that were originally installed with 1.x

[Jacopo Rota]

As per discourse post https://discourse.maas.io/t/upgrade-to-3-4-fails/7761

snap refresh --channel=3.4 maas

results in SQL error:

Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running migrations:
  Applying metadataserver.0034_use_builtin_json_field... OK
  Applying maasserver.0291_rdns_hostnames_as_array... OK
  Applying maasserver.0292_use_builtin_json_field...Traceback (most recent call last):
  File "/snap/maas/32469/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.DatatypeMismatch: default for column "extra" cannot be cast automatically to type jsonb

and

  Applying maasserver.0302_big_auto_field...Traceback (most recent call last):
  File "/snap/maas/32469/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.NumericValueOutOfRange: setval: value 0 is out of bounds for sequence "maasserver_vlan_id_seq" (1..9223372036854775807)

[Jacopo Rota (r00ta)]

From the user's database

```
psql (14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))
Type "help" for help.

maasdb=# \d maasserver_bootresourcefile;
                                         Table "public.maasserver_bootresourcefile"
     Column | Type | Collation | Nullable | Default
-----------------+--------------------------+-----------+----------+---------------------------------------------------------
 id | integer | | not null | nextval('maasserver_bootresourcefile_id_seq'::regclass)
 created | timestamp with time zone | | not null |
 updated | timestamp with time zone | | not null |
 resource_set_id | integer | | not null |
 largefile_id | integer | | not null |
 filename | character varying(255) | | not null |
 filetype | character varying(20) | | not null | 'tgz'::character varying
 extra | text | | not null | '""'::text
Indexes:
    "maasserver_bootresourcefile_pkey" PRIMARY KEY, btree (id)
    "maasserver_bootresourcefi_resource_set_id_1bdfd52f7cc6de80_uniq" UNIQUE CONSTRAINT, btree (resource_set_id, filename)
    "maasserver_bootresourcefile_largefile_id" btree (largefile_id)
    "maasserver_bootresourcefile_resource_set_id" btree (resource_set_id)
Foreign-key constraints:
    "largefile_id_refs_id_6192daf196cc0427" FOREIGN KEY (largefile_id) REFERENCES maasserver_largefile(id) DEFERRABLE INITIALLY DEFERRED
    "resource_set_id_refs_id_5740a0afd6adefa0" FOREIGN KEY (resource_set_id) REFERENCES maasserver_bootresourceset(id) DEFERRABLE INITIALLY DEFERRED

I'd say that the migration fails to cast the extra column to jsonb due to its default value '""'::text. The fix would be to specify how this column should be migrated - i.e. should should not just cast the column to jsonb.
```

However, from a plain 3.2/stable installation I could not manage to get the same default value for the column

```
ubuntu@maas-testmigration:~$ sudo maas-test-db.psql
psql (12.4)
Type "help" for help.

maasdb=# select * from ^C
maasdb=# \d maasserver_bootresourcefile;
                                         Table "public.maasserver_bootresourcefile"
     Column | Type | Collation | Nullable | Default
-----------------+--------------------------+-----------+----------+---------------------------------------------------------
 id | integer | | not null | nextval('maasserver_bootresourcefile_id_seq'::regclass)
 created | timestamp with time zone | | not null |
 updated | timestamp with time zone | | not null |
 filename | character varying(255) | | not null |
 filetype | character varying(20) | | not null |
 extra | text | | not null |
 largefile_id | integer | | not null |
 resource_set_id | integer | | not null |
```...

[Jacopo Rota (r00ta)]

The user reports that this is an old MAAS installation coming from the 2.x serie.

I tested up to MAAS 2.5 and still could not get the default column value set to '""'::text.

Anyhow, in order to reproduce this one could simply run

```
ALTER TABLE maasserver_bootresourcefile ALTER COLUMN extra SET DEFAULT '""';
```

and then try to refresh the snap.

The simple temporary workaround is to run

```
ALTER TABLE maasserver_bootresourcefile ALTER COLUMN extra SET DEFAULT NULL;
```

instead.

[Jacopo Rota (r00ta)]

Looking at the 0292_use_builtin_json_field.py migration file we the following impacted tables

- bootresource
- bootresourcefile
- bootsourcecache
- config
- interface
- notification

From the user's db we can see that `config`, `notification` and `bootsourcecache` are not impacted. So the final workaround is the following:

```
# migrate maasserver_bootresource
ALTER TABLE maasserver_bootresource ALTER COLUMN extra SET DEFAULT NULL;

# migrate maasserver_bootresourcefile
ALTER TABLE maasserver_bootresourcefile ALTER COLUMN extra SET DEFAULT NULL;

# migrate maasserver_interface
ALTER TABLE maasserver_interface ALTER COLUMN ipv4_params SET DEFAULT NULL;
ALTER TABLE maasserver_interface ALTER COLUMN ipv6_params SET DEFAULT NULL;
ALTER TABLE maasserver_interface ALTER COLUMN params SET DEFAULT NULL;
```

[Jacopo Rota (r00ta)]

I managed to reproduce the errors. In order to reproduce this we have to go back to MAAS 1.9, where the migrations were applied by South. In MAAS 2.0 the team ported the migrations to the new builtin tool provided by django and apparently South and Django were treating the default column value in a different way.