Error on request (58) node.check_power

[Andres Rodriguez]

Manually check the power state of a Machine works successfully, however, when trying to check the power state in the WEbUI, this always keeps failing, regardless of it being reliable in the CLI. This seems to be affecting other states.

Initially thought this was only affecnting nodes in Releasing state, but also affecting nodes in ready state.

2015-04-14 08:08:36 [-] 127.0.0.1 - - [14/Apr/2015:00:08:36 +0000] "GET /MAAS/rpc/ HTTP/1.1" 200 146 "-" "Twisted PageGetter"
2015-04-14 08:08:45 [maas.websockets.node] ERROR: bronze-shake.local: Timed out waiting for power response in Node.power_state
2015-04-14 08:08:18 [-] Error on request (58) node.check_power: [26/170]
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 763, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 404, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 300, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 151, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 399, in call_within_transaction
            return func_within_txn(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/maasserver/websockets/base.py", line 324, in execute
            return m(params)
          File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/node.py", line 518, in check_power
            update_power_state(state).wait(15)
          File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 217, in wait
            result = self._result(timeout)
          File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 195, in _result
            raise TimeoutError()
        crochet._eventloop.TimeoutError:

[Andres Rodriguez (andreserl)]

Further logging:

Apr 14 08:05:12 trusty-maas9 maas.power: [ERROR] Error changing power state (off) of node: vague-joke.local (node-3860140e-dc69-11e4-839c-00163e
11c887)
Apr 14 08:05:12 trusty-maas9 maas.power: [ERROR] Error changing power state (off) of node: tragic-color.local (node-3f7a3ecc-dc69-11e4-8842-0016
3e11c887)
Apr 14 08:05:13 trusty-maas9 maas.power: [ERROR] Error changing power state (off) of node: valuable-pies.local (node-292180b8-dc69-11e4-850f-001
63e11c887)
Apr 14 08:05:13 trusty-maas9 maas.power: [ERROR] Error changing power state (off) of node: vibrant-balance.local (node-34600472-dc69-11e4-b8d9-0
0163e11c887)
Apr 14 08:05:13 trusty-maas9 maas.power: [ERROR] Error changing power state (off) of node: yearly-rest.local (node-432ae2ec-dc69-11e4-8842-00163
e11c887)

==> /var/log/maas/regiond.log <==
          File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/node.py", line 518, in check_power
            update_power_state(state).wait(15)
          File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 217, in wait
            result = self._result(timeout)
          File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 195, in _result
            raise TimeoutError()
        crochet._eventloop.TimeoutError: