MAAS 1.9 fails to upgrade in some scenarios

Bug #1516722 reported by Mark W Wenning on 2015-11-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Critical
Blake Rouse
1.9
Critical
Blake Rouse

Bug Description

System is Trusty in a VM, 1 CPU, 4 GB Ram, 38 GB disk.

sudo add-apt-repository ppa:maas/next
sudo apt-get update
sudo apt-get install maas

Install proceeded until the following error:
Error in migration: maasserver:0182_initial_networking_layout
Traceback (most recent call last):
  File "/usr/sbin/maas-region-admin", line 68, in <module>
    main()
  File "/usr/sbin/maas-region-admin", line 64, in main
    run()
  File "/usr/sbin/maas-region-admin", line 57, in run
    management.execute_from_command_line()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/south/management/commands/migrate.py", line 107, in handle
    ignore_ghosts = ignore_ghosts,
  File "/usr/lib/python2.7/dist-packages/south/migration/__init__.py", line 219, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 235, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 310, in migrate_many
    result = self.migrate(migration, database)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 133, in migrate
    result = self.run(migration)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 107, in run
    return self.run_migration(migration)
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 81, in run_migration
    migration_function()
  File "/usr/lib/python2.7/dist-packages/south/migration/migrators.py", line 57, in <lambda>
    return (lambda: direction(orm))
  File "/usr/lib/python2.7/dist-packages/maasserver/migrations/0182_initial_networking_layout.py", line 24, in forwards
    node.set_initial_networking_configuration()
  File "/usr/lib/python2.7/dist-packages/maasserver/models/node.py", line 2194, in set_initial_networking_configuration
    boot_interface.force_auto_or_dhcp_link()
  File "/usr/lib/python2.7/dist-packages/maasserver/models/interface.py", line 882, in force_auto_or_dhcp_link
    for subnet in self.vlan.subnet_set.all():
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 318, in __get__
    rel_obj = qs.get()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 301, in get
    num = len(clone)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column maasserver_vlan.mtu does not exist
LINE 1: ...erver_vlan"."vid", "maasserver_vlan"."fabric_id", "maasserve...
                                                             ^
=====
Version info:
ubuntu@MAAS-Server:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-===================================================-============-===============================================================================
iU maas 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server all-in-one metapackage
ii maas-cert-server 0.2.11-0~40~ubuntu14.04.1 all Ubuntu certification support files for MAAS server
ii maas-cli 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS command line API tool
ii maas-cluster-controller 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server cluster controller
ii maas-common 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server common files
ii maas-dhcp 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS DHCP server
ii maas-dns 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS DNS server
ii maas-proxy 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS Caching Proxy
iF maas-region-controller 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server complete region controller
ii maas-region-controller-min 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS Server minimum region controller
ii python-django-maas 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server Django web framework
ii python-maas-client 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS python API client
ii python-maas-provisioningserver 1.9.0~rc1+bzr4496-0ubuntu1~trusty1 all MAAS server provisioning libraries
ubuntu@MAAS-Server:~$

===
tar of /var/log/maass/* included.

Related branches

Mark W Wenning (mwenning) wrote :
Changed in maas:
importance: Undecided → Critical
milestone: none → 1.9.0
Changed in maas:
status: New → Triaged
Changed in maas:
assignee: nobody → Mike Pontillo (mpontillo)
status: Triaged → In Progress
summary: - MAAS 1.9 fails install
+ MAAS 1.9 fails to upgrade in some escenarios
Changed in maas:
assignee: Mike Pontillo (mpontillo) → Blake Rouse (blake-rouse)
Changed in maas:
milestone: 1.9.0 → next
Mike Pontillo (mpontillo) wrote :

In case anyone needs a workaround before the final fix for this, I want to point out that this only affects the upgrade path if there are nodes in READY state. Therefore, a possible workaround is:

"sudo maas-region-admin dbshell"

followed by

"update maasserver_node set status=8 where status=4;"

This process will mark any READY nodes BROKEN. It's likely that nodes will need to be recommissioned after doing this before they can be deployed.

summary: - MAAS 1.9 fails to upgrade in some escenarios
+ MAAS 1.9 fails to upgrade in some scenarios
Changed in maas:
status: In Progress → Fix Committed
Mark W Wenning (mwenning) wrote :

I can verify that the workaround works.

Changed in maas:
status: Fix Committed → Fix Released
milestone: next → none
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments