Activity log for bug #1443709

Date Who What changed Old value New value Message
2015-04-14 00:11:17 Andres Rodriguez bug added bug
2015-04-14 00:12:32 Andres Rodriguez description 2015-04-14 08:08:36 [-] 127.0.0.1 - - [14/Apr/2015:00:08:36 +0000] "GET /MAAS/rpc/ HTTP/1.1" 200 146 "-" "Twisted PageGetter" 2015-04-14 08:08:45 [maas.websockets.node] ERROR: bronze-shake.local: Timed out waiting for power response in Node.power_state 2015-04-14 08:08:18 [-] Error on request (58) node.check_power: [26/170] Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 404, in call_within_transaction return func_outside_txn(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 300, in retrier return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 151, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 399, in call_within_transaction return func_within_txn(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/maasserver/websockets/base.py", line 324, in execute return m(params) File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/node.py", line 518, in check_power update_power_state(state).wait(15) File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 217, in wait result = self._result(timeout) File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 195, in _result raise TimeoutError() crochet._eventloop.TimeoutError: Manually check the power state of a Machine works successfully, however, when trying to check the power state in the WEbUI, this always keeps failing, regardless of it being reliable in the CLI. This seems to be affecting other states. The particular issue is that this node is in 'Releasing' State. 2015-04-14 08:08:36 [-] 127.0.0.1 - - [14/Apr/2015:00:08:36 +0000] "GET /MAAS/rpc/ HTTP/1.1" 200 146 "-" "Twisted PageGetter" 2015-04-14 08:08:45 [maas.websockets.node] ERROR: bronze-shake.local: Timed out waiting for power response in Node.power_state 2015-04-14 08:08:18 [-] Error on request (58) node.check_power: [26/170]         Traceback (most recent call last):           File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap             self.__bootstrap_inner()           File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner             self.run()           File "/usr/lib/python2.7/threading.py", line 763, in run             self.__target(*self.__args, **self.__kwargs)         --- <exception caught here> ---           File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker             result = context.call(ctx, function, *args, **kwargs)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext             return self.currentContext().callWithContext(ctx, func, *args, **kw)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext             return func(*args,**kw)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 404, in call_within_transaction             return func_outside_txn(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 300, in retrier             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 151, in wrapper             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 399, in call_within_transaction             return func_within_txn(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/websockets/base.py", line 324, in execute             return m(params)           File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/node.py", line 518, in check_power             update_power_state(state).wait(15)           File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 217, in wait             result = self._result(timeout)           File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 195, in _result             raise TimeoutError()         crochet._eventloop.TimeoutError:
2015-04-14 00:14:03 Andres Rodriguez description Manually check the power state of a Machine works successfully, however, when trying to check the power state in the WEbUI, this always keeps failing, regardless of it being reliable in the CLI. This seems to be affecting other states. The particular issue is that this node is in 'Releasing' State. 2015-04-14 08:08:36 [-] 127.0.0.1 - - [14/Apr/2015:00:08:36 +0000] "GET /MAAS/rpc/ HTTP/1.1" 200 146 "-" "Twisted PageGetter" 2015-04-14 08:08:45 [maas.websockets.node] ERROR: bronze-shake.local: Timed out waiting for power response in Node.power_state 2015-04-14 08:08:18 [-] Error on request (58) node.check_power: [26/170]         Traceback (most recent call last):           File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap             self.__bootstrap_inner()           File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner             self.run()           File "/usr/lib/python2.7/threading.py", line 763, in run             self.__target(*self.__args, **self.__kwargs)         --- <exception caught here> ---           File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker             result = context.call(ctx, function, *args, **kwargs)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext             return self.currentContext().callWithContext(ctx, func, *args, **kw)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext             return func(*args,**kw)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 404, in call_within_transaction             return func_outside_txn(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 300, in retrier             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 151, in wrapper             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 399, in call_within_transaction             return func_within_txn(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/websockets/base.py", line 324, in execute             return m(params)           File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/node.py", line 518, in check_power             update_power_state(state).wait(15)           File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 217, in wait             result = self._result(timeout)           File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 195, in _result             raise TimeoutError()         crochet._eventloop.TimeoutError: Manually check the power state of a Machine works successfully, however, when trying to check the power state in the WEbUI, this always keeps failing, regardless of it being reliable in the CLI. This seems to be affecting other states. Initially thought this was only affecnting nodes in Releasing state, but also affecting nodes in ready state. 2015-04-14 08:08:36 [-] 127.0.0.1 - - [14/Apr/2015:00:08:36 +0000] "GET /MAAS/rpc/ HTTP/1.1" 200 146 "-" "Twisted PageGetter" 2015-04-14 08:08:45 [maas.websockets.node] ERROR: bronze-shake.local: Timed out waiting for power response in Node.power_state 2015-04-14 08:08:18 [-] Error on request (58) node.check_power: [26/170]         Traceback (most recent call last):           File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap             self.__bootstrap_inner()           File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner             self.run()           File "/usr/lib/python2.7/threading.py", line 763, in run             self.__target(*self.__args, **self.__kwargs)         --- <exception caught here> ---           File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker             result = context.call(ctx, function, *args, **kwargs)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext             return self.currentContext().callWithContext(ctx, func, *args, **kw)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext             return func(*args,**kw)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 404, in call_within_transaction             return func_outside_txn(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 300, in retrier             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 151, in wrapper             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 399, in call_within_transaction             return func_within_txn(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner             return func(*args, **kwargs)           File "/usr/lib/python2.7/dist-packages/maasserver/websockets/base.py", line 324, in execute             return m(params)           File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/node.py", line 518, in check_power             update_power_state(state).wait(15)           File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 217, in wait             result = self._result(timeout)           File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 195, in _result             raise TimeoutError()         crochet._eventloop.TimeoutError:
2015-04-14 00:14:54 Andres Rodriguez maas: importance Undecided Critical
2015-04-14 00:14:59 Andres Rodriguez maas: status New Triaged
2015-04-14 00:15:01 Andres Rodriguez maas: milestone 1.8.0
2015-04-14 11:44:31 Blake Rouse branch linked lp:~rvb/maas/huge-threadpool
2015-04-14 11:44:38 Blake Rouse maas: status Triaged Fix Committed
2015-06-22 19:28:54 Andres Rodriguez maas: status Fix Committed Fix Released