internal server error when deleting a node

[Blake Rouse]

Unable to delete a node in the WebUI. This node was marked declared. This error causes the node to not even be removed.

ERROR 2014-09-11 10:44:17,087 django.request Internal Server Error: /MAAS/nodes/node-6622ec04-39c1-11e4-b6a1-00163ebe2ae4/delete/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/edit.py", line 257, in post
    return self.delete(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/views/__init__.py", line 82, in delete
    self.object.delete()
  File "/usr/lib/python2.7/dist-packages/maasserver/models/node.py", line 989, in delete
    self.delete_host_maps(set().union(static_ips, leased_ips))
  File "/usr/lib/python2.7/dist-packages/maasserver/models/node.py", line 1017, in delete_host_maps
    raise MultipleFailures(*remove_host_maps_failures)
MultipleFailures: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
]

[Blake Rouse (blake-rouse)]

I also have seen this error, once the node is ready.

2014-09-11 11:34:15-0400 [ClusterClient,client] Amp server or network failure unhandled by client application. Dropping connection! To avoid, add errbacks to ALL remote commands!
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 913, in ampBoxReceived
     self._commandReceived(box)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 892, in _commandReceived
     deferred.addCallback(self._safeEmit)
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 306, in addCallback
     callbackKeywords=kw)
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 295, in addCallbacks
     self._runCallbacks()
 --- <exception caught here> ---
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 924, in _safeEmit
     aBox._sendTo(self.boxSender)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 577, in _sendTo
     proto.sendBox(self)
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 2153, in sendBox
     self.transport.write(box.serialize())
   File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 551, in serialize
     "Unicode value for key %r not allowed: %r" % (k, v))
 exceptions.TypeError: Unicode value for key '_error_code' not allowed: u'CannotRemoveHostMap'

[Blake Rouse (blake-rouse)]

This is also why "juju bootstrap" is failing.

I have test it and juju results in:

ubuntu@maas-test:~$ juju bootstrap
Launching instance
WARNING picked arbitrary tools &{1.20.5-trusty-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.20.5-trusty-amd64.tgz 7bb999e3cd2776148dcc4dafbd4af39655715187716f64d78f7a81cc42fd3fff 8224472}
^[[1;2CERROR bootstrap failed: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"power_type": ["\nThe cluster controller for this node is not responding; power type\nvalidation is not available.\nUnable to get RPC connection for cluster 'maas'"]})
Bootstrap failed, destroying environment
ERROR cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"power_type": ["\nThe cluster controller for this node is not responding; power type\nvalidation is not available.\nUnable to get RPC connection for cluster 'maas'"]})

This is because the following happens in django, which drops the connection to the cluster, causing the power_type validation to fail.

INFO 2014-09-11 11:36:54,231 twisted RegionServer connection lost (HOST:IPv4Address(TCP, '127.0.0.1', 34533) PEER:IPv4Address(TCP, '127.0.0.1', 49858))
ERROR 2014-09-11 11:36:54,232 maasserver ################################ Exception: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone$
>: Connection was closed cleanly.
] ################################
ERROR 2014-09-11 11:36:54,232 maasserver Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/decorators/vary.py", line 19, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/piston/resource.py", line 167, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/usr/lib/python2.7/dist-packages/piston/resource.py", line 165, in __call__
    result = meth(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/api/support.py", line 189, in dispatch
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/api/nodes.py", line 279, in start
    [system_id], request.user, user_data=user_data)
  File "/usr/lib/python2.7/dist-packages/maasserver/models/node.py", line 494, in start_nodes
    raise MultipleFailures(*update_host_maps_failures)
MultipleFailures: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
]

ERROR 2014-09-11 11:36:54,326 maasserver Unable to get RPC connection for cluster 'maas'
ERROR 2014-09-11 11:36:54,328 maasserver Unable to get RPC connection for cluster 'maas'
ERROR 2014-09-11 11:36:54,332 maasserver Unable to get RPC connection for cluster 'maas'
ERROR 2014-09-11 11:36:54,340 maasserver Unable to get RPC connection for cluster 'maas'
ERROR 2014-09-11 11:36:54,513 maasserver Unable to get RPC connection for cluster 'maas'
...many more...
ERROR 2014-0...

[Graham Binns (gmb)]

The error in comment #1 is because the errors CannotRemoveHostmap and CannotCreateHostmap are declared incorrectly — they should be declared as ascii strings, not unicode. Fixing that now…

[Gavin Panella (allenap)]

> The error in comment #1 is because the errors CannotRemoveHostmap and
> CannotCreateHostmap are declared incorrectly — they should be declared
> as ascii strings, not unicode. Fixing that now…

I think this is what's causing the connection to drop, which is
unfortunately masking the underlying error. Can you try again once
Graham's fixes have landed?

[Blake Rouse (blake-rouse)]

I guess that unicode error was causing the other errors. I am not getting this error anymore with Graham's branch merged. I am going to mark this as Fix Committed.