MAAS

[2.2 RC2] upgrade from Beta 4: Traceback error "Not enough values to unpack" on most UI tabs

Bug #1683440 reported by Jeff Lane  on 2017-04-17

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	MAAS	Fix Released	Critical	Lee Trager	MAAS 2.2.0rc3

Bug Description

bladernr@critical-maas:~$ dpkg -l |grep maas|awk '{print $2" "$3}'
maas 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-cert-server 0.2.30-0~76~ubuntu16.04.1
maas-cli 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-common 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-dhcp 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-dns 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-proxy 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-rack-controller 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-region-api 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
maas-region-controller 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
python3-django-maas 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
python3-maas-client 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1
python3-maas-provisioningserver 2.2.0~rc2+bzr5983-0ubuntu1~16.04.1

Just updated from 2.2.0-beta4 to 2.2.0-rc2 on Xenial. I logged out and back into the web UI and the dashboard now shows this error (see attached screenshot). No amount of reloading will make it disappear.

I have also determined that this affects just about the entire UI, including Dashboard, Nodes, Images, Pods, Subnets, DNS tabs.

The Zones tab loads (I only have the default zone anyway). The Settings tab returns an "Internal Server Error" page.

This is the traceback that appears in the logs:

2017-04-17 13:55:13 maasserver.websockets.protocol: [critical] Error on request (13) general.osinfo: not enough values to unpack (expected 2, got 1)

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 842, in worker
    return target()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 686, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 504, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/general.py", line 104, in osinfo
    releases = list_all_usable_releases(osystems)
  File "/usr/lib/python3/dist-packages/maasserver/utils/osystems.py", line 70, in list_all_usable_releases
    osystem, release = resource.name.split('/')
builtins.ValueError: not enough values to unpack (expected 2, got 1)
2017-04-17 13:55:14 maasserver.websockets.protocol: [critical] Error on request (24) config.list: not enough values to unpack (expected 2, got 1)

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 842, in worker
    return target()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 686, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 504, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/config.py", line 67, in list
    return self._include_choices(config_keys)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/config.py", line 45, in _include_choices
    self._include_choice(config_key)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/config.py", line 38, in _include_choice
    config_field = get_config_field(config_key['name'])
  File "/usr/lib/python3/dist-packages/maasserver/forms/settings.py", line 549, in get_config_field
    return conf['form'](**kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/forms/settings.py", line 102, in make_default_distro_series_field
    [default_usable_os])[default_osystem]
  File "/usr/lib/python3/dist-packages/maasserver/utils/osystems.py", line 70, in list_all_usable_releases
    osystem, release = resource.name.split('/')
builtins.ValueError: not enough values to unpack (expected 2, got 1)

See original description

Tags:

Related branches

lp:~ltrager/maas/lp1683440

Merged into lp:~maas-committers/maas/trunk at revision 5993

Mike Pontillo (community): Approve on 2017-04-17

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-04-17:

maas-dashboard-bug.png Edit (13.9 KiB, image/png)

tags:

added: hwcert-server

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-04-17:

1683340-maas-logs.tgz Edit (51.8 KiB, application/x-tar)

Additionally, here are the actual logs:

description:

updated

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-04-17:

I just rebooted my MAAS server and this still happens, so whatever happened during that upgrade, it is well and truly b0rked. MAAS is completely unusable.

summary:

- 2.2 RC2 Dashboard error "Not enough values to unpack"
+ 2.2 RC2 upgrade from Beta 4: Traceback error "Not enough values to
+ unpack" on most UI tabs

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-04-17: Re: 2.2 RC2 upgrade from Beta 4: Traceback error "Not enough values to unpack" on most UI tabs

Just to be sure, I rebooted a second time and it's still DOA.

Blake Rouse (blake-rouse) on 2017-04-17

Changed in maas:
status:	New → Triaged
importance:	Undecided → Critical
milestone:	none → 2.2.0rc3
summary:	- 2.2 RC2 upgrade from Beta 4: Traceback error "Not enough values to + [2.2 RC2] upgrade from Beta 4: Traceback error "Not enough values to unpack" on most UI tabs

Revision history for this message

Rod Smith (rodsmith) wrote on 2017-04-17:

FWIW, I just upgraded from 2.2.0-rc1 to 2.2.0-rc2 and I've encountered this problem, too.

Lee Trager (ltrager) on 2017-04-17

Changed in maas:
assignee:	nobody → Lee Trager (ltrager)

Revision history for this message

Lee Trager (ltrager) wrote on 2017-04-17:

Could you please post the output of maas <profile> boot-resources read

Changed in maas:
status:	Triaged → Incomplete

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-04-17:

Download full text (10.0 KiB)

bladernr@critical-maas:~$ maas admin boot-resources read
Success.
Machine-readable output follows:
[
    {
        "id": 62,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/62/",
        "name": "centos/centos66",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 61,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/61/",
        "name": "centos/centos70",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 64,
        "resource_uri": "/MAAS/api/2.0/boot-resources/64/",
        "name": "grub-efi-signed/uefi",
        "type": "Synced",
        "architecture": "amd64/generic"
    },
    {
        "id": 65,
        "resource_uri": "/MAAS/api/2.0/boot-resources/65/",
        "name": "grub-efi/uefi",
        "type": "Synced",
        "architecture": "arm64/generic"
    },
    {
        "id": 66,
        "resource_uri": "/MAAS/api/2.0/boot-resources/66/",
        "name": "grub-ieee1275/open-firmware",
        "type": "Synced",
        "architecture": "ppc64el/generic"
    },
    {
        "id": 63,
        "resource_uri": "/MAAS/api/2.0/boot-resources/63/",
        "name": "pxelinux/pxe",
        "type": "Synced",
        "architecture": "i386/generic"
    },
    {
        "id": 35,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/35/",
        "name": "ubuntu/trusty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t",
        "architecture": "amd64/hwe-t"
    },
    {
        "id": 32,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/32/",
        "name": "ubuntu/trusty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u",
        "architecture": "amd64/hwe-u"
    },
    {
        "id": 33,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/33/",
        "name": "ubuntu/trusty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v",
        "architecture": "amd64/hwe-v"
    },
    {
        "id": 34,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/34/",
        "name": "ubuntu/trusty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w",
        "architecture": "amd64/hwe-w"
    },
    {
        "id": 31,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/31/",
        "name": "ubuntu/trusty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,hwe-x",
        "architecture": "amd64/hwe-x"
    },
    {
        "id": 36,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/36/",
        "name": "ubuntu/trusty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,hwe-x",
        "architecture": "amd64/hwe-x-lowlatency"
    },
    {
        "id": 8,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/8/",
        "name": "ubuntu/xenial",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04",
        "architecture": "amd64/ga-16.04"
    },
    {
        "id": 5,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/5/",
        "name": "ubuntu/xenial",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04",
        "architecture": "amd64/ga-16.04-lowlatency"
    },
    {
        "id": 7,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/7/",
        "name": "ubuntu/xenial",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04",
        "architecture": "amd64/hwe-16.04"
    },
    {
        "id": 68,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/68/",
        "name": "ubuntu/xenial",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,hwe-16.04,hwe-16.10",
        "architecture": "amd64/hwe-16.04-edge"
    },
    {
        "id": 6,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/6/",
        "name": "ubuntu/xenial",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04",
        "architecture": "amd64/hwe-16.04-lowlatency"
    },
    {
        "id": 67,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/67/",
        "name": "ubuntu/xenial",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,hwe-16.04,hwe-16.10",
        "architecture": "amd64/hwe-16.04-lowlatency-edge"
    },
    {
        "id": 50,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/50/",
        "name": "ubuntu/yakkety",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,ga-16.10",
        "architecture": "amd64/ga-16.10"
    },
    {
        "id": 56,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/56/",
        "name": "ubuntu/yakkety",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,ga-16.10",
        "architecture": "amd64/ga-16.10-lowlatency"
    },
    {
        "id": 57,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/57/",
        "name": "ubuntu/yakkety",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,ga-16.10",
        "architecture": "amd64/hwe-16.10"
    },
    {
        "id": 60,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/60/",
        "name": "ubuntu/yakkety",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,ga-16.10",
        "architecture": "amd64/hwe-16.10-lowlatency"
    },
    {
        "id": 59,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/59/",
        "name": "ubuntu/zesty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,ga-16.10,ga-17.04",
        "architecture": "amd64/ga-17.04"
    },
    {
        "id": 58,
        "type": "Synced",
        "resource_uri": "/MAAS/api/2.0/boot-resources/58/",
        "name": "ubuntu/zesty",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t,hwe-u,hwe-v,hwe-w,ga-16.04,ga-16.10,ga-17.04",
        "architecture": "amd64/ga-17.04-lowlatency"
    },
    {
        "id": 48,
        "resource_uri": "/MAAS/api/2.0/boot-resources/48/",
        "type": "Uploaded",
        "title": "Ubuntu 12.04.1 (amd64)",
        "name": "u12.04.1_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 47,
        "resource_uri": "/MAAS/api/2.0/boot-resources/47/",
        "type": "Uploaded",
        "title": "Ubuntu 12.04.2 (amd64)",
        "name": "u12.04.2_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 46,
        "resource_uri": "/MAAS/api/2.0/boot-resources/46/",
        "type": "Uploaded",
        "title": "Ubuntu 12.04.3 (amd64)",
        "name": "u12.04.3_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 45,
        "resource_uri": "/MAAS/api/2.0/boot-resources/45/",
        "type": "Uploaded",
        "title": "Ubuntu 12.04.4 (amd64)",
        "name": "u12.04.4_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 44,
        "resource_uri": "/MAAS/api/2.0/boot-resources/44/",
        "type": "Uploaded",
        "title": "Ubuntu 12.04.5 (amd64)",
        "name": "u12.04.5_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 49,
        "resource_uri": "/MAAS/api/2.0/boot-resources/49/",
        "type": "Uploaded",
        "title": "Ubuntu 12.04 GA (amd64)",
        "name": "u12.04_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 42,
        "resource_uri": "/MAAS/api/2.0/boot-resources/42/",
        "type": "Uploaded",
        "title": "Ubuntu 14.04.1 GA (amd64)",
        "name": "u14.04.1_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 41,
        "resource_uri": "/MAAS/api/2.0/boot-resources/41/",
        "type": "Uploaded",
        "title": "Ubuntu 14.04.2 GA (amd64)",
        "name": "u14.04.2_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 40,
        "resource_uri": "/MAAS/api/2.0/boot-resources/40/",
        "type": "Uploaded",
        "title": "Ubuntu 14.04.3 GA (amd64)",
        "name": "u14.04.3_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 39,
        "resource_uri": "/MAAS/api/2.0/boot-resources/39/",
        "type": "Uploaded",
        "title": "Ubuntu 14.04.4 GA (amd64)",
        "name": "u14.04.4_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 38,
        "resource_uri": "/MAAS/api/2.0/boot-resources/38/",
        "type": "Uploaded",
        "title": "Ubuntu 14.04.5 GA (amd64)",
        "name": "u14.04.5_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 43,
        "resource_uri": "/MAAS/api/2.0/boot-resources/43/",
        "type": "Uploaded",
        "title": "Ubuntu 14.04 GA (amd64)",
        "name": "u14.04_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 9,
        "resource_uri": "/MAAS/api/2.0/boot-resources/9/",
        "type": "Uploaded",
        "title": "Ubuntu 16.04.1 GA (amd64)",
        "name": "u16.04.1_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    },
    {
        "id": 37,
        "resource_uri": "/MAAS/api/2.0/boot-resources/37/",
        "type": "Uploaded",
        "title": "Ubuntu 16.04 GA (amd64)",
        "name": "u16.04_amd64",
        "subarches": "generic",
        "architecture": "amd64/generic"
    }
]

Changed in maas:
status:	Incomplete → Confirmed

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-04-17:

This also breaks maas on the CLI:
bladernr@critical-maas:~$ maas admin machine deploy -d pbrbtr distro_series=xenial
500 INTERNAL SERVER ERROR

         Connection: close
       Content-Type: text/plain; charset=utf-8
               Date: Mon, 17 Apr 2017 20:11:21 GMT
             Server: TwistedWeb/16.0.0
             Status: 500
  Transfer-Encoding: chunked
               Vary: Cookie
    X-Frame-Options: SAMEORIGIN

not enough values to unpack (expected 2, got 1)

Lee Trager (ltrager) on 2017-04-17

Changed in maas:
status:	Confirmed → In Progress

MAAS Lander (maas-lander) on 2017-04-18

Changed in maas:
status:	In Progress → Fix Committed

Cory Vogel (vogelc) on 2017-04-18

Changed in maas:
status:	Fix Committed → Fix Released

MAAS Lander (maas-lander) on 2017-04-18

Changed in maas:
status:	Fix Released → Fix Committed

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-04-19:

Just to confirm, I upgraded from dailybuilds and this does appear to have fixed the issue, I can now deploy nodes and perform other tasks.

Andres Rodriguez (andreserl) on 2017-04-27

Changed in maas:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.