Failed to power on nodes on SM15k

Bug #1515769 reported by Kenny Ho on 2015-11-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
High
Newell Jensen
1.9
High
Newell Jensen
Trunk
High
Newell Jensen

Bug Description

When trying to commission a node on the SM15000 using MAAS 1.9 beta 2, I got error: "Failed to power on node - Node could not be powered on: Failed talking to node's BMC for an unknown reason." Further investigation show this trace in /var/log/maas/clusterd.log:

2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power could not be turned on.
                Traceback (most recent call last):
                  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
                    self._startRunCallbacks(result)
                  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
                    self._runCallbacks()
                  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
                    current.result = callback(current.result, *args, **kw)
                  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
                    _inlineCallbacks(r, g, deferred)
                --- <exception caught here> ---
                  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
                    result = result.throwExceptionIntoGenerator(g)
                  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
                    return g.throw(self.type, self.value, self.tb)
                  File "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py", line 287, in change_power_state
                    system_id, hostname, power_type, power_change, context)
                  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
                    result = result.throwExceptionIntoGenerator(g)
                  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
                    return g.throw(self.type, self.value, self.tb)
                  File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/__init__.py", line 276, in perform_power
                    power_func, system_id, **kwargs)
                  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
                    result = context.call(ctx, function, *args, **kwargs)
                  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
                    return self.currentContext().callWithContext(ctx, func, *args, **kw)
                  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
                    return func(*args,**kw)
                  File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/seamicro.py", line 84, in power_on
                    self._power('on', **kwargs)
                  File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/seamicro.py", line 80, in _power
                    ip, username, password, server_id, power_change=power_change)
                  File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/hardware/seamicro.py", line 332, in power_control_seamicro15k_v2
                    server = api.servers.get(server_id)
                  File "/usr/local/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 71, in get
                    "/servers/%s" % base.getid(server))
                  File "/usr/local/lib/python2.7/dist-packages/seamicroclient/base.py", line 89, in _get
                    _resp, body = self.api.client.get(url)
                  File "/usr/local/lib/python2.7/dist-packages/seamicroclient/client.py", line 173, in get
                    return self._cs_request(url, 'GET', **kwargs)
                  File "/usr/local/lib/python2.7/dist-packages/seamicroclient/client.py", line 158, in _cs_request
                    method,**kwargs)
                  File "/usr/local/lib/python2.7/dist-packages/seamicroclient/client.py", line 140, in _time_request
                    resp, body = self.request(url, method, **kwargs)
                  File "/usr/local/lib/python2.7/dist-packages/seamicroclient/client.py", line 134, in request
                    raise exceptions.from_response(resp, body, url, method)
                seamicroclient.exceptions.NotFound: Not Found: servers/None/0 (HTTP 404)

After digging into the code, turns out there's a bug in provisioningserver/drivers/power/seamicro.py
(the system_id was not passed into _power()). Fixes below:

def power_on(self, system_id, **kwargs):
        """Power on SeaMicro node."""
        self._power('on', system_id=system_id, **kwargs)

def power_off(self, system_id, **kwargs):
        """Power off SeaMicro node."""
        self._power('off', system_id=system_id, **kwargs)

Related branches

Changed in maas:
milestone: none → 1.9.0
Changed in maas:
assignee: nobody → Newell Jensen (newell-jensen)
Changed in maas:
status: New → Triaged
importance: Undecided → High
Newell Jensen (newell-jensen) wrote :

Kenny,

Do you have the "System ID" saved in the Power section for the Node in the WebUI?

Download full text (5.8 KiB)

Yes I did. It just wasn't passed into the function for the extract
params. The fix is included at the bottom of the bug report.

Regards,
Kenny
On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
wrote:

> Kenny,
>
> Do you have the "System ID" saved in the Power section for the Node in
> the WebUI?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1515769
>
> Title:
> Failed to power on nodes on SM15k
>
> Status in MAAS:
> Triaged
>
> Bug description:
> When trying to commission a node on the SM15000 using MAAS 1.9 beta 2,
> I got error: "Failed to power on node - Node could not be powered on:
> Failed talking to node's BMC for an unknown reason." Further
> investigation show this trace in /var/log/maas/clusterd.log:
>
> 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> could not be turned on.
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in
> callback
> self._startRunCallbacks(result)
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in
> _startRunCallbacks
> self._runCallbacks()
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in
> _runCallbacks
> current.result = callback(current.result, *args,
> **kw)
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in
> gotResult
> _inlineCallbacks(r, g, deferred)
> --- <exception caught here> ---
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in
> _inlineCallbacks
> result = result.throwExceptionIntoGenerator(g)
> File
> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in
> throwExceptionIntoGenerator
> return g.throw(self.type, self.value, self.tb)
> File
> "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py", line
> 287, in change_power_state
> system_id, hostname, power_type, power_change,
> context)
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in
> _inlineCallbacks
> result = result.throwExceptionIntoGenerator(g)
> File
> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in
> throwExceptionIntoGenerator
> return g.throw(self.type, self.value, self.tb)
> File
> "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/__init__.py",
> line 276, in perform_power
> power_func, system_id, **kwargs)
> File
> "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191,
> in _worker
> result = context.call(ctx, function, *args, **kwargs)
> File
> "/u...

Read more...

Newell Jensen (newell-jensen) wrote :
Download full text (12.0 KiB)

Kenny,

I understand and see that. However, the context (kwargs are gathered from
the context of the user data that is saved from the power parameters of the
UI). Could you do me a favor and print out the system_id and kwargs that
are passed into the original power_on or power_off functions and list them
here?

Thanks

On Fri, Nov 13, 2015 at 11:23 AM, Kenny Ho <email address hidden> wrote:

> Yes I did. It just wasn't passed into the function for the extract
> params. The fix is included at the bottom of the bug report.
>
> Regards,
> Kenny
> On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
> wrote:
>
> > Kenny,
> >
> > Do you have the "System ID" saved in the Power section for the Node in
> > the WebUI?
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1515769
> >
> > Title:
> > Failed to power on nodes on SM15k
> >
> > Status in MAAS:
> > Triaged
> >
> > Bug description:
> > When trying to commission a node on the SM15000 using MAAS 1.9 beta 2,
> > I got error: "Failed to power on node - Node could not be powered on:
> > Failed talking to node's BMC for an unknown reason." Further
> > investigation show this trace in /var/log/maas/clusterd.log:
> >
> > 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> > could not be turned on.
> > Traceback (most recent call last):
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382,
> in
> > callback
> > self._startRunCallbacks(result)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490,
> in
> > _startRunCallbacks
> > self._runCallbacks()
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577,
> in
> > _runCallbacks
> > current.result = callback(current.result, *args,
> > **kw)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155,
> in
> > gotResult
> > _inlineCallbacks(r, g, deferred)
> > --- <exception caught here> ---
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097,
> in
> > _inlineCallbacks
> > result = result.throwExceptionIntoGenerator(g)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389,
> in
> > throwExceptionIntoGenerator
> > return g.throw(self.type, self.value, self.tb)
> > File
> > "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py",
> line
> > 287, in change_power_state
> > system_id, hostname, power_type, power_change,
> > context)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097,
> in
> > _inlineCallbacks
> > result = result.throwExceptionIntoGenerator(g)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/python/fa...

Kenny Ho (y2kenny) wrote :
Download full text (19.4 KiB)

Sure (I blanked out some IP/mac address/username password, etc.):
2015-11-13 14:13:18-0600 [ClusterClient,client] power_on
2015-11-13 14:13:18-0600 [ClusterClient,client] {u'power_control':
u'restapi2', u'ipmipower': u'/usr/sbin/ipmipower', u'ipmitool':
u'/usr/bin/ipmitool', u'fence_cdu': u'/usr/sbin/fence_cdu', u'power_id':
u'node-*******', u'power_user': u'*******', u'username': u'',
u'ipmi_chassis_config': u'/usr/sbin/ipmi-chassis-config', u'power_address':
u'*********', u'mac_address': u'*******', u'power_driver': u'',
u'boot_mode': u'pxe', u'power_off_mode': u'', u'power_pass': u'*******',
u'ipmi_config': u'ipmi.conf'}
2015-11-13 14:13:29-0600 [ClusterClient,client] power_off
2015-11-13 14:13:29-0600 [ClusterClient,client] {u'power_control':
u'restapi2', u'ipmipower': u'/usr/sbin/ipmipower', u'ipmitool':
u'/usr/bin/ipmitool', u'fence_cdu': u'/usr/sbin/fence_cdu', u'power_id':
u'node-*******', u'power_user': u'*******', u'username': u'',
u'ipmi_chassis_config': u'/usr/sbin/ipmi-chassis-config', u'power_address':
u'*********', u'mac_address': u'*******', u'power_driver': u'',
u'boot_mode': u'pxe', u'power_off_mode': u'hard', u'power_pass':
u'*******', u'ipmi_config': u'ipmi.conf'}

On Fri, Nov 13, 2015 at 2:57 PM, Newell Jensen <email address hidden>
wrote:

> Kenny,
>
> I understand and see that. However, the context (kwargs are gathered from
> the context of the user data that is saved from the power parameters of the
> UI). Could you do me a favor and print out the system_id and kwargs that
> are passed into the original power_on or power_off functions and list them
> here?
>
> Thanks
>
> On Fri, Nov 13, 2015 at 11:23 AM, Kenny Ho <email address hidden> wrote:
>
> > Yes I did. It just wasn't passed into the function for the extract
> > params. The fix is included at the bottom of the bug report.
> >
> > Regards,
> > Kenny
> > On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
> > wrote:
> >
> > > Kenny,
> > >
> > > Do you have the "System ID" saved in the Power section for the Node in
> > > the WebUI?
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1515769
> > >
> > > Title:
> > > Failed to power on nodes on SM15k
> > >
> > > Status in MAAS:
> > > Triaged
> > >
> > > Bug description:
> > > When trying to commission a node on the SM15000 using MAAS 1.9 beta
> 2,
> > > I got error: "Failed to power on node - Node could not be powered on:
> > > Failed talking to node's BMC for an unknown reason." Further
> > > investigation show this trace in /var/log/maas/clusterd.log:
> > >
> > > 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> > > could not be turned on.
> > > Traceback (most recent call last):
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382,
> > in
> > > callback
> > > self._startRunCallbacks(result)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490,
> > in
> > > _startRunCallbacks
> > > self._runC...

Blake Rouse (blake-rouse) wrote :
Download full text (18.5 KiB)

Newell,

The issue is that the power parameters for sm15k include a field named
system_id. Instead of passing all the parameters as a **kwarg just pass it
as a dictionary to all power types. That will prevent the collision.
On Nov 13, 2015 3:05 PM, "Newell Jensen" <email address hidden>
wrote:

> Kenny,
>
> I understand and see that. However, the context (kwargs are gathered from
> the context of the user data that is saved from the power parameters of the
> UI). Could you do me a favor and print out the system_id and kwargs that
> are passed into the original power_on or power_off functions and list them
> here?
>
> Thanks
>
> On Fri, Nov 13, 2015 at 11:23 AM, Kenny Ho <email address hidden> wrote:
>
> > Yes I did. It just wasn't passed into the function for the extract
> > params. The fix is included at the bottom of the bug report.
> >
> > Regards,
> > Kenny
> > On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
> > wrote:
> >
> > > Kenny,
> > >
> > > Do you have the "System ID" saved in the Power section for the Node in
> > > the WebUI?
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1515769
> > >
> > > Title:
> > > Failed to power on nodes on SM15k
> > >
> > > Status in MAAS:
> > > Triaged
> > >
> > > Bug description:
> > > When trying to commission a node on the SM15000 using MAAS 1.9 beta
> 2,
> > > I got error: "Failed to power on node - Node could not be powered on:
> > > Failed talking to node's BMC for an unknown reason." Further
> > > investigation show this trace in /var/log/maas/clusterd.log:
> > >
> > > 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> > > could not be turned on.
> > > Traceback (most recent call last):
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382,
> > in
> > > callback
> > > self._startRunCallbacks(result)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490,
> > in
> > > _startRunCallbacks
> > > self._runCallbacks()
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577,
> > in
> > > _runCallbacks
> > > current.result = callback(current.result, *args,
> > > **kw)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
> 1155,
> > in
> > > gotResult
> > > _inlineCallbacks(r, g, deferred)
> > > --- <exception caught here> ---
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
> 1097,
> > in
> > > _inlineCallbacks
> > > result = result.throwExceptionIntoGenerator(g)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389,
> > in
> > > throwExceptionIntoGenerator
> > > return g.throw(self.type, self.value, self.tb)
> > > File
> > > "/usr/lib/python2.7...

Mike Pontillo (mpontillo) wrote :

Ah, perhaps using (*args, **kwargs) would fix it; is the system_id being passed as a non-named parameter somewhere?

Mike Pontillo (mpontillo) wrote :

(Just saw Blake's comment; I like that better. Explicit is better than implicit.)

Newell Jensen (newell-jensen) wrote :

Okay we can either change the way the current implementation is done (as mentioned above) or we can "correct" the power schema that sm15k currently has by changing "System ID" to "Server ID" for the sm15k (which is what it should be). This would need a data migration. I had talked to Blake about changing the implementation, and already have branch for this. However, on further thought and seeing that sm15k is the only hardware that has system_id as one of its power parameters, it might be better to just change this power power parameter. Thoughts from the MAAS team?

Changed in maas:
status: Triaged → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers