"Internal server error." on cluster page in webUI

[Mike Rushton]

This is an upgrade from MAAS 1.5 to 1.7

I had added an additional cluster controller via the MAAS CLI.

I then added the daily images repository via the MAAS CLI.

At times I will get the message that one of the clusters is down. We have determined we no longer need the second cluster controller so I have attempted to remove it from the webUI. The above error message is intermittent, showing about 9/10 times. The one time I was able to get in, I attempted to remove the 2nd cluster controller but got the internal server error and now not sure if it went through. I have also powered off the second cluster controller.

logs from hitting refresh on the clusters page:

1.7.0+bzr3299-0ubuntu1~trusty

Internal server error.

==> /var/log/maas/pserv.log <==
2014-11-19 14:08:56-0500 [Uninitialized] ClusterClient connection established (HOST:IPv4Address(TCP, '127.0.0.1', 56236) PEER:IPv4Address(TCP, u'127.0.0.1', 44257))
2014-11-19 14:08:56-0500 [Uninitialized] ClusterClient connection established (HOST:IPv4Address(TCP, '127.0.0.1', 56236) PEER:IPv4Address(TCP, u'127.0.0.1', 44257))
2014-11-19 14:08:56-0500 [ClusterClient,client] Event-loop 'lucuma:pid=750' authenticated.
2014-11-19 14:08:56-0500 [ClusterClient,client] Event-loop 'lucuma:pid=750' authenticated.
2014-11-19 14:08:56-0500 [ClusterClient,client] Event-loop 'lucuma:pid=751' authenticated.
2014-11-19 14:08:56-0500 [ClusterClient,client] Event-loop 'lucuma:pid=751' authenticated.
2014-11-19 14:08:57-0500 [ClusterClient,client] Cluster 'bc161c5c-530d-4bd0-9b59-5ce2b6da87d8' registered (via lucuma:pid=750).
2014-11-19 14:08:57-0500 [ClusterClient,client] Cluster 'bc161c5c-530d-4bd0-9b59-5ce2b6da87d8' registered (via lucuma:pid=750).
2014-11-19 14:08:57-0500 [ClusterClient,client] Cluster 'bc161c5c-530d-4bd0-9b59-5ce2b6da87d8' registered (via lucuma:pid=751).
2014-11-19 14:08:57-0500 [ClusterClient,client] Cluster 'bc161c5c-530d-4bd0-9b59-5ce2b6da87d8' registered (via lucuma:pid=751).
2014-11-19 14:10:20-0500 [ClusterClient,client] Amp server or network failure unhandled by client application. Dropping connection! To avoid, add errbacks to ALL remote commands!
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 913, in ampBoxReceived
     self._commandReceived(box)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 892, in _commandReceived
     deferred.addCallback(self._safeEmit)
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 306, in addCallback
     callbackKeywords=kw)
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 295, in addCallbacks
     self._runCallbacks()
 --- <exception caught here> ---
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 924, in _safeEmit
     aBox._sendTo(self.boxSender)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 577, in _sendTo
     proto.sendBox(self)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 2153, in sendBox
     self.transport.write(box.serialize())
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 555, in serialize
     raise TooLong(False, True, v, k)
 twisted.protocols.amp.TooLong:
2014-11-19 14:10:20-0500 [ClusterClient,client] ClusterClient connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 56236) PEER:IPv4Address(TCP, u'127.0.0.1', 44257))
2014-11-19 14:10:20-0500 [ClusterClient,client] ClusterClient connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 56236) PEER:IPv4Address(TCP, u'127.0.0.1', 44257))
2014-11-19 14:10:20-0500 [ClusterClient,client] Logged OOPS id OOPS-e42aafed2a186ca1d96ba2a7a34affde: TooLong:

==> /var/log/maas/maas-django.log <==
INFO 2014-11-19 14:10:20,517 twisted RegionServer connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 44257) PEER:IPv4Address(TCP, '127.0.0.1', 56236))
ERROR 2014-11-19 14:10:20,539 django.request Internal Server Error: /MAAS/clusters/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 137, in get_response
    response = response.render()
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 82, in rendered_content
    content = template.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 203, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 155, in render
    return self.render_template(self.template, context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 137, in render_template
    output = template.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 504, in render
    six.iteritems(self.extra_context)])
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 585, in resolve
    obj = self.var.resolve(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 735, in resolve
    value = self._resolve_lookup(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 789, in _resolve_lookup
    current = current()
  File "/usr/lib/python2.7/dist-packages/maasserver/models/nodegroup.py", line 274, in get_state
    images = get_boot_images(self)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 143, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/clusterrpc/boot_images.py", line 93, in get_boot_images
    return call.wait(30).get("images")
  File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 219, in wait
    result.raiseException()
  File "<string>", line 2, in raiseException
ConnectionDone: Connection was closed cleanly.

[Mike Rushton (leftyfb)]

removing the source for the daily images:

ubuntu@lucuma:~$ maas ubuntu boot-sources read
[
    {
        "url": "http://maas.ubuntu.com/images/ephemeral-v2/releases/",
        "keyring_data": "",
        "resource_uri": "/MAAS/api/1.0/boot-sources/1/",
        "keyring_filename": "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg",
        "id": 1
    },
    {
        "url": "http://maas.ubuntu.com/images/ephemeral-v2/daily/",
        "keyring_data": "",
        "resource_uri": "/MAAS/api/1.0/boot-sources/2/",
        "keyring_filename": "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg",
        "id": 2
    }

ubuntu@lucuma:~$ maas ubuntu boot-source delete 2

This has now allowed access to the clusters page.

[Mike Rushton (leftyfb)]

After deleting the second cluster controller and removing the additional daily images source, I have since lost the ability to connect to the /clusters/ page. Not noticing a pattern as of yet.

[Julian Edwards (julian-edwards)]

On Thursday 20 Nov 2014 17:44:44 you wrote:
> removing the source for the daily images:
>
> ubuntu@lucuma:~$ maas ubuntu boot-sources read
> [
> {
> "url": "http://maas.ubuntu.com/images/ephemeral-v2/releases/",
> "keyring_data": "",
> "resource_uri": "/MAAS/api/1.0/boot-sources/1/",
> "keyring_filename":
> "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg", "id": 1
> },
> {
> "url": "http://maas.ubuntu.com/images/ephemeral-v2/daily/",
> "keyring_data": "",
> "resource_uri": "/MAAS/api/1.0/boot-sources/2/",
> "keyring_filename":
> "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg", "id": 2
> }
>
> ubuntu@lucuma:~$ maas ubuntu boot-source delete 2
>
> This has now allowed access to the clusters page.

Have you got the traceback from this particular one please?

[Mike Rushton (leftyfb)]

Unfortunately, I do not. There have been too many changes to the server to try to fix things that it is not in the same state. Also still not fully functional. See https://bugs.launchpad.net/maas/+bug/1395120