maas upgrade 0072_remove_ipmi_autodetect fails

[Blake Rouse]

Performing upgrade of maas to package version 1.5+bzr2204-0ubuntu1 database migration failed. Migration failed on 0072_remove_ipmi_autodetect.

[Blake Rouse (blake-rouse)]

Upgrade log:

http://paste.ubuntu.com/7200204/

[Julian Edwards (julian-edwards)]

This does not happen when I upgrade from precise to trusty.

Have you been running with unreleased packages in the daily PPA?

If so you will need to kill your database and re-create it (or edit it yourself). We will often edit migrations in-between releases (where release is equivalent to "uploaded to trusty") as nobody will have run it yet (apart from brave developers!)

[Julian Edwards (julian-edwards)]

Alternatively if precise -> trusty and saucy -> trusty all upgrade OK, I don't know if we care much about this bug, but can you confirm either way please. Thanks!

[Blake Rouse (blake-rouse)]

This was an upgrade of the trusty package to the new trusty package. This looks to be a data migration, could it be more related to actual data in the database.

I was able to get it to upgrade successfully by ignoring this migration, since its only to force ipmi v2.0.

[Julian Edwards (julian-edwards)]

Oh I hadn't realised it was a data migration. Can you provide the results of this please:

sudo maas dbshell --install
> select * from maasserver_node;

It should help track down where the bug is.

Thank you!

[Jeroen T. Vermeulen (jtv)]

Typo: that option is --installed, not --install. Maybe I should just make it automatic so we don't have to worry about this...

[Blake Rouse (blake-rouse)]

Here is the output, I only selected the power_type and power_parameters, as that is what the data migration is looking at. I replaced the passwords and the last octet of the ip address with *.

 id | power_type | power_parameters
----+------------+-------------------------------------------------------------------------------------------------------------------------------
 33 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
 12 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
 34 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
 28 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
 30 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
 36 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
  4 | ipmi | {"power_driver": "LAN", "power_address": "10.245.0.*", "power_pass": "*", "power_user": "maas"}
  8 | ipmi | {"power_driver": "", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
 16 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
  5 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
 15 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
 26 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
 37 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", power_pass": "*", "power_user": "maas"}
 24 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
  7 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
 31 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
 27 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "maas"}
  9 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
  3 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
 13 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
 11 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}
 32 | ipmi | {"power_driver": "LAN_2_0", "power_address": "192.168.220....

[Raphaël Badin (rvb)]

I didn't manage to recreate the problem yet… I installed the package 1.5+bzr2227-0ubuntu1, created a node with power_type = 'ipmi' and power_parameters = {"power_driver": "", "power_address": "192.168.220.*", "power_pass": "*", "power_user": "Administrator"}.

Then I ran:
sudo maas-region-admin migrate maasserver 0071_drop_after_commissioning_action
(to rewind the migrations)
sudo maas-region-admin migrate maasserver 0072_remove_ipmi_autodetect
(to run the 0072_remove_ipmi_autodetect migration)

No crash, and I can see the node's power_parameters field being updated (i.e. after the migration has run, node.power_parameters['power_driver'] is 'LAN_2_0').

[Julian Edwards (julian-edwards)]

I think we'll have to leave this incomplete for now as nobody can recreate it.

Raphaël, it might be worth catching exceptions in the migration to be on the safe side, so we don't leave package installations hanging.

[Raphaël Badin (rvb)]

> Raphaël, it might be worth catching exceptions in the migration to be on the safe side, so we don't leave package installations hanging.

Sounds like a good idea. At least the installation of the package won't be broken if this re-appears.

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]

[Bill Nash (billn)]

I can duplicate this on two identical systems, both upgraded from precise through to trusty:

root@xamot:/var/www# apt-get install maas
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  maas-region-controller
The following NEW packages will be installed:
  maas maas-region-controller
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/6,754 B of archives.
After this operation, 93.2 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Selecting previously unselected package maas-region-controller.
(Reading database ... 128110 files and directories currently installed.)
Preparing to unpack .../maas-region-controller_1.5.2+bzr2282-0ubuntu0.2_all.deb ...
Unpacking maas-region-controller (1.5.2+bzr2282-0ubuntu0.2) ...
Selecting previously unselected package maas.
Preparing to unpack .../maas_1.5.2+bzr2282-0ubuntu0.2_all.deb ...
Unpacking maas (1.5.2+bzr2282-0ubuntu0.2) ...
Setting up maas-region-controller (1.5.2+bzr2282-0ubuntu0.2) ...
Considering dependency proxy for proxy_http:
Module proxy already enabled
Module proxy_http already enabled
Module expires already enabled
Module wsgi already enabled
rsyslog stop/waiting
rsyslog start/running, process 12044
squid-deb-proxy stop/waiting
squid-deb-proxy start/running, process 12100
 * Restarting message broker rabbitmq-server [ OK ]
Changing password for user "maas_longpoll" ...
...done.
Changing password for user "maas_workers" ...
...done.
 * Restarting PostgreSQL 9.3 database server [ OK ]
dbconfig-common: writing config to /etc/dbconfig-common/maas-region-controller.conf
creating postgres user maas: success.
verifying creation of user: success.
creating database maasdb: success.
verifying database maasdb exists: success.
dbconfig-common: flushing administrative password
Syncing...
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table piston_nonce
Creating table piston_consumer
Creating table piston_token
Creating table south_migrationhistory
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

Synced:
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.messages
 > django.contrib.staticfiles
 > piston
 > south

Not synced (use migrations):
 - maasserver
 - metadataserver
(use ./manage.py migrate to migrate these)
UnknownMigration: Migration 'maasserver:0072_remove_ipmi_autodetect' probably doesn't exist.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/south/migration/base.py", line 297, in migration
  ...

[David J. Collamore (djcollamore)]

I get the exact same error log as Bill Nash on a brand new install of Trusty.

[Samir Ibradžić (sibradzic)]

As *Bill Nash* and others may have noticed, the fix for this bug introduces a strange regression when installing maas from scratch., during *maas-region-admin migrate metadataserver* phase initiated by maas-region-controller postinst script . The following workaround fixed it for me;

1. install *python-django-maas* before everything else,
2. comment out

from django.db.utils import ProgrammingError

in /usr/share/pyshared/maasserver/migrations/0072_remove_ipmi_autodetect.py, which is a part of python-django-maas package.

3. install the rest of maass