Activity log for bug #1508741

Date Who What changed Old value New value Message
2015-10-22 01:34:56 Andres Rodriguez bug added bug
2015-10-22 01:35:02 Andres Rodriguez maas: milestone 1.9.0
2015-10-22 01:35:03 Andres Rodriguez maas: importance Undecided Critical
2015-10-26 15:43:59 Gavin Panella summary MAAS ipmi power does not handle timeouts correclt IPMI driver does not handle timeouts correctly
2015-10-26 15:44:17 Gavin Panella description MAAS is unable to reach the BMC's. Howeve,r instead of correctly handling the timeout, it is erroring out. The machines shoudl show a RED for power error in the UI, but it is not the case. They still show the status before this issue (which was on) 2015-10-22 09:31:04+0800 [ClusterClient,client] late-women: Power could not be turned off. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback self._startRunCallbacks(result) File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks self._runCallbacks() File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py", line 287, in change_power_state system_id, hostname, power_type, power_change, context) File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/__init__.py", line 276, in perform_power power_func, system_id, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 166, in power_off self._issue_ipmi_command('off', **kwargs) File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 144, in _issue_ipmi_command ipmi_chassis_config_command, power_change, power_address, env) File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 82, in _issue_ipmi_chassis_config_command "Failed to power %s %s: %s" % (change, address, stderr)) provisioningserver.drivers.power.PowerFatalError: Failed to power off 192.168.4.27: /usr/sbin/ipmi-chassis-config: connection timeout MAAS is unable to reach the BMCs. However instead of correctly handling the timeout, it is erroring out. The machines shoudld show a RED for power error in the UI, but it is not the case. They still show the status before this issue (which was on). 2015-10-22 09:31:04+0800 [ClusterClient,client] late-women: Power could not be turned off.         Traceback (most recent call last):           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback             self._startRunCallbacks(result)           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks             self._runCallbacks()           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks             current.result = callback(current.result, *args, **kw)           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult             _inlineCallbacks(r, g, deferred)         --- <exception caught here> ---           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks             result = result.throwExceptionIntoGenerator(g)           File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator             return g.throw(self.type, self.value, self.tb)           File "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py", line 287, in change_power_state             system_id, hostname, power_type, power_change, context)           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks             result = result.throwExceptionIntoGenerator(g)           File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator             return g.throw(self.type, self.value, self.tb)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/__init__.py", line 276, in perform_power             power_func, system_id, **kwargs)           File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker             result = context.call(ctx, function, *args, **kwargs)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext             return self.currentContext().callWithContext(ctx, func, *args, **kw)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext             return func(*args,**kw)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 166, in power_off             self._issue_ipmi_command('off', **kwargs)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 144, in _issue_ipmi_command             ipmi_chassis_config_command, power_change, power_address, env)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 82, in _issue_ipmi_chassis_config_command             "Failed to power %s %s: %s" % (change, address, stderr))         provisioningserver.drivers.power.PowerFatalError: Failed to power off 192.168.4.27: /usr/sbin/ipmi-chassis-config: connection timeout
2015-10-26 15:46:06 Gavin Panella description MAAS is unable to reach the BMCs. However instead of correctly handling the timeout, it is erroring out. The machines shoudld show a RED for power error in the UI, but it is not the case. They still show the status before this issue (which was on). 2015-10-22 09:31:04+0800 [ClusterClient,client] late-women: Power could not be turned off.         Traceback (most recent call last):           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback             self._startRunCallbacks(result)           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks             self._runCallbacks()           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks             current.result = callback(current.result, *args, **kw)           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult             _inlineCallbacks(r, g, deferred)         --- <exception caught here> ---           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks             result = result.throwExceptionIntoGenerator(g)           File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator             return g.throw(self.type, self.value, self.tb)           File "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py", line 287, in change_power_state             system_id, hostname, power_type, power_change, context)           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks             result = result.throwExceptionIntoGenerator(g)           File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator             return g.throw(self.type, self.value, self.tb)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/__init__.py", line 276, in perform_power             power_func, system_id, **kwargs)           File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker             result = context.call(ctx, function, *args, **kwargs)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext             return self.currentContext().callWithContext(ctx, func, *args, **kw)           File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext             return func(*args,**kw)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 166, in power_off             self._issue_ipmi_command('off', **kwargs)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 144, in _issue_ipmi_command             ipmi_chassis_config_command, power_change, power_address, env)           File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/ipmi.py", line 82, in _issue_ipmi_chassis_config_command             "Failed to power %s %s: %s" % (change, address, stderr))         provisioningserver.drivers.power.PowerFatalError: Failed to power off 192.168.4.27: /usr/sbin/ipmi-chassis-config: connection timeout MAAS is unable to reach the BMCs. However instead of correctly handling the timeout, it is erroring out. The machines shoudld show a RED for power error in the UI, but this is not the case. They still show the status before this issue (which was on). 2015-10-22 09:31:04+0800 [ClusterClient,client] late-women: Power could not be turned off. Traceback (most recent call last): File ".../twisted/internet/defer.py", line 382, in callback self._startRunCallbacks(result) File ".../twisted/internet/defer.py", line 490, in _startRunCallbacks self._runCallbacks() File ".../twisted/internet/defer.py", line 577, in _runCallbacks current.result = callback(current.result, *args, **kw) File ".../twisted/internet/defer.py", line 1155, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File ".../twisted/internet/defer.py", line 1097, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File ".../twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File ".../provisioningserver/power/change.py", line 287, in change_power_state system_id, hostname, power_type, power_change, context) File ".../twisted/internet/defer.py", line 1097, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File ".../twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File ".../provisioningserver/drivers/power/__init__.py", line 276, in perform_power power_func, system_id, **kwargs) File ".../twisted/python/threadpool.py", line 191, in _worker result = context.call(ctx, function, *args, **kwargs) File ".../twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File ".../twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File ".../provisioningserver/drivers/power/ipmi.py", line 166, in power_off self._issue_ipmi_command('off', **kwargs) File ".../provisioningserver/drivers/power/ipmi.py", line 144, in _issue_ipmi_command ipmi_chassis_config_command, power_change, power_address, env) File ".../provisioningserver/drivers/power/ipmi.py", line 82, in _issue_ipmi_chassis_config_command "Failed to power %s %s: %s" % (change, address, stderr)) provisioningserver.drivers.power.PowerFatalError: Failed to power off 192.168.4.27: /usr/sbin/ipmi-chassis-config: connection timeout
2015-10-26 15:58:40 Gavin Panella maas: status New Invalid
2015-10-26 16:35:13 Andres Rodriguez maas: status Invalid New
2015-10-26 16:35:15 Andres Rodriguez maas: status New Triaged
2015-10-26 16:56:22 Gavin Panella description MAAS is unable to reach the BMCs. However instead of correctly handling the timeout, it is erroring out. The machines shoudld show a RED for power error in the UI, but this is not the case. They still show the status before this issue (which was on). 2015-10-22 09:31:04+0800 [ClusterClient,client] late-women: Power could not be turned off. Traceback (most recent call last): File ".../twisted/internet/defer.py", line 382, in callback self._startRunCallbacks(result) File ".../twisted/internet/defer.py", line 490, in _startRunCallbacks self._runCallbacks() File ".../twisted/internet/defer.py", line 577, in _runCallbacks current.result = callback(current.result, *args, **kw) File ".../twisted/internet/defer.py", line 1155, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File ".../twisted/internet/defer.py", line 1097, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File ".../twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File ".../provisioningserver/power/change.py", line 287, in change_power_state system_id, hostname, power_type, power_change, context) File ".../twisted/internet/defer.py", line 1097, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File ".../twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File ".../provisioningserver/drivers/power/__init__.py", line 276, in perform_power power_func, system_id, **kwargs) File ".../twisted/python/threadpool.py", line 191, in _worker result = context.call(ctx, function, *args, **kwargs) File ".../twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File ".../twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File ".../provisioningserver/drivers/power/ipmi.py", line 166, in power_off self._issue_ipmi_command('off', **kwargs) File ".../provisioningserver/drivers/power/ipmi.py", line 144, in _issue_ipmi_command ipmi_chassis_config_command, power_change, power_address, env) File ".../provisioningserver/drivers/power/ipmi.py", line 82, in _issue_ipmi_chassis_config_command "Failed to power %s %s: %s" % (change, address, stderr)) provisioningserver.drivers.power.PowerFatalError: Failed to power off 192.168.4.27: /usr/sbin/ipmi-chassis-config: connection timeout MAAS is unable to reach the BMCs. However instead of correctly handling the timeout, it is erroring out. The machines should show a RED for power error in the UI, but this is not the case. They still show the status before this issue (which was on). The machines are actually off.
2015-10-26 19:09:19 Ricardo Bánffy maas: assignee Ricardo Bánffy (rbanffy)
2015-10-28 17:04:58 Andres Rodriguez maas: status Triaged In Progress
2015-10-30 18:40:49 Ricardo Bánffy branch linked lp:~rbanffy/maas/1508741-ipmi-fail-timeout
2016-01-05 17:45:19 Andres Rodriguez maas: milestone 1.9.0 1.9.1
2016-02-04 13:25:23 Gavin Panella maas: status In Progress Triaged
2016-02-04 13:25:26 Gavin Panella maas: assignee Ricardo Bánffy (rbanffy)
2016-02-04 22:09:30 Gavin Panella maas: status Triaged Incomplete
2016-03-01 19:45:40 Andres Rodriguez maas: milestone 1.9.1 1.9.2
2016-04-21 20:18:16 Newell Jensen maas: status Incomplete Triaged
2016-04-21 20:28:39 Newell Jensen maas: assignee Newell Jensen (newell-jensen)
2016-04-21 20:30:09 Newell Jensen branch unlinked lp:~rbanffy/maas/1508741-ipmi-fail-timeout
2016-04-21 22:01:11 Newell Jensen maas: status Triaged In Progress
2016-04-22 20:17:39 Launchpad Janitor branch linked lp:~newell-jensen/maas/fix-1508741-1.9
2016-04-22 21:31:27 Newell Jensen branch unlinked lp:~newell-jensen/maas/fix-1508741-1.9
2016-04-22 22:21:28 Launchpad Janitor branch linked lp:~newell-jensen/maas/fix-1508741-1.9
2016-04-23 00:18:07 Launchpad Janitor branch linked lp:~newell-jensen/maas/fix-1508741-1.9
2016-04-25 21:16:10 Newell Jensen maas: status In Progress Fix Committed
2016-04-28 18:41:27 Andres Rodriguez maas: status Fix Committed Fix Released
2016-05-18 13:27:57 Andres Rodriguez branch linked lp:~newell-jensen/maas/update-fix-1508741-1.9
2016-05-18 13:30:24 Andres Rodriguez branch linked lp:~newell-jensen/maas/ipmi-regex-return-output-when-no-match
2016-05-18 13:30:54 Andres Rodriguez nominated for series maas/2.0
2016-05-18 13:30:54 Andres Rodriguez bug task added maas/2.0
2016-05-18 13:37:01 Andres Rodriguez maas/2.0: assignee Gavin Panella (allenap)
2016-05-18 13:37:23 Andres Rodriguez branch linked lp:~allenap/maas/ipmi-power-confusion--bug-1560830
2016-05-18 13:37:29 Andres Rodriguez maas/2.0: importance Undecided Critical
2016-05-18 13:37:32 Andres Rodriguez maas/2.0: status New In Progress
2016-05-20 17:30:01 Gavin Panella maas/2.0: status In Progress Fix Committed
2017-05-17 00:52:41 Newell Jensen maas/2.0: status Fix Committed Fix Released