MAAS 1.9 fails to upgrade in some scenarios

Bug #1516722 reported by Mark W Wenning
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse
1.9
Fix Released
Critical
Blake Rouse

Bug Description

System is Trusty in a VM, 1 CPU, 4 GB Ram, 38 GB disk.

sudo add-apt-repository ppa:maas/next
sudo apt-get update
sudo apt-get install maas

Install proceeded until the following error:
Error in migration: maasserver:0182_initial_networking_layout
Traceback (most recent call last):
  File "/usr/sbin/maas-region-admin", line 68, in <module>
    main()
  File "/usr/sbin/maas-region-admin", line 64, in main
    run()
  File "/usr/sbin/maas-region-admin", line 57, in run
    management.execute_from_command_line()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/south/management/commands/migrate.py", line 107, in handle
    ignore_ghosts = ignore_ghosts,
  File "/usr/lib/python2.7/dist-packages/south/migration/__init__.py", line 219, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 235, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 310, in migrate_many
    result = self.migrate(migration, database)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 133, in migrate
    result = self.run(migration)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 107, in run
    return self.run_migration(migration)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 81, in run_migration
    migration_function()
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 57, in <lambda>
    return (lambda: direction(orm))
  File "/usr/lib/python2.7/dist-packages/maasserver/migrations/0182_initial_networking_layout.py", line 24, in forwards
    node.set_initial_networking_configuration()
  File "/usr/lib/python2.7/dist-packages/maasserver/models/node.py", line 2194, in set_initial_networking_configuration
    boot_interface.force_auto_or_dhcp_link()
  File "/usr/lib/python2.7/dist-packages/maasserver/models/interface.py", line 882, in force_auto_or_dhcp_link
    for subnet in self.vlan.subnet_set.all():
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 318, in __get__
    rel_obj = qs.get()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 301, in get
    num = len(clone)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column maasserver_vlan.mtu does not exist
LINE 1: ...erver_vlan"."vid", "maasserver_vlan"."fabric_id", "maasserve...
                                                             ^
=====
Version info:
ubuntu@MAAS-Server:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-===================================================-============-===============================================================================
iU maas 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server all-in-one metapackage
ii maas-cert-server 0.2.11-0~40~ubuntu14.04.1 all Ubuntu certification support files for MAAS server
ii maas-cli 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS command line API tool
ii maas-cluster-controller 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server cluster controller
ii maas-common 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server common files
ii maas-dhcp 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS DHCP server
ii maas-dns 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS DNS server
ii maas-proxy 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS Caching Proxy
iF maas-region-controller 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server complete region controller
ii maas-region-controller-min 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS Server minimum region controller
ii python-django-maas 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server Django web framework
ii python-maas-client 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS python API client
ii python-maas-provisioningserver 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server provisioning libraries
ubuntu@MAAS-Server:~$

===
tar of /var/log/maass/* included.

Related branches

Revision history for this message
Mark W Wenning (mwenning) wrote :
Changed in maas:
importance: Undecided → Critical
milestone: none → 1.9.0
Changed in maas:
status: New → Triaged
Changed in maas:
assignee: nobody → Mike Pontillo (mpontillo)
status: Triaged → In Progress
summary: - MAAS 1.9 fails install
+ MAAS 1.9 fails to upgrade in some escenarios
Changed in maas:
assignee: Mike Pontillo (mpontillo) → Blake Rouse (blake-rouse)
Changed in maas:
milestone: 1.9.0 → next
Revision history for this message
Mike Pontillo (mpontillo) wrote :

In case anyone needs a workaround before the final fix for this, I want to point out that this only affects the upgrade path if there are nodes in READY state. Therefore, a possible workaround is:

"sudo maas-region-admin dbshell"

followed by

"update maasserver_node set status=8 where status=4;"

This process will mark any READY nodes BROKEN. It's likely that nodes will need to be recommissioned after doing this before they can be deployed.

summary: - MAAS 1.9 fails to upgrade in some escenarios
+ MAAS 1.9 fails to upgrade in some scenarios
Changed in maas:
status: In Progress → Fix Committed
Revision history for this message
Mark W Wenning (mwenning) wrote :

I can verify that the workaround works.

Changed in maas:
status: Fix Committed → Fix Released
milestone: next → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.