Power interface not working for Redfish hardware

[Madhuri Kumari]

I have a RSD node which I registered to Ironic. But the power interface is not working for Redfish hardwares. Below are my observations seeing ironic-conductor logs:

1. There is error message in log which says "During sync_power_state, could not get power state for node fdcdf9db-71b7-4525-9ceb-99437499f350, attempt 1 of 3. Error: 'int' object has no attribute 'find'."

2. Also I am seeing, the ipmitool command in log for the same redfish power on request which seems odd to me. Because we are talking to sushy to power on/off our node.
Running cmd (subprocess): ipmitool -I lanplus -H 10.223.197.220 -L ADMINISTRATOR -p 6232 -U admin -R 12 -N 5 -f /tmp/tmpN0Va_9 power status

3. I tried to add some debug messages to ironic to see how the power request URI looks like for the Redfish hardware and it seems the request didn't go through to it.

[Lucas Alvares Gomes (lucasagomes)]

Thanks for opening this bug Madhuri!

I reproduce this problem in my virtual environment built with [0]. Can you please let me know which version of Ironic are you using ? Is it master ? Or <some version> + cherry-picked redfish driver ?

[0] https://review.openstack.org/445529

Cheers,
Lucas

[Lucas Alvares Gomes (lucasagomes)]

urgh typo... I mean, I *could not* reproduce this problem in my virtual environment ...

[Madhuri Kumari (madhuri-rai07)]

Lucas,
I am using master. And have a real RSD node.
Also I tried sushy manually and it was working fine.

[Madhuri Kumari (madhuri-rai07)]

ubuntu@ubuntu:/opt/stack/ironic$ ironic node-show test-node
+------------------------+-------------------------------------------------------------------------+
| Property | Value |
+------------------------+-------------------------------------------------------------------------+
| boot_interface | |
| chassis_uuid | None |
| clean_step | {} |
| console_enabled | False |
| console_interface | |
| created_at | 2017-05-03T11:16:42+00:00 |
| deploy_interface | |
| driver | redfish |
| driver_info | {u'redfish_verify_ca': False, u'redfish_system_id': 173, |
| | u'redfish_address': u'https://127.0.0.1:8443/redfish/v1/', |
| | u'redfish_username': u'admin', u'redfish_password': u'******'} |
| driver_internal_info | {} |
| extra | {} |
| inspect_interface | |
| inspection_finished_at | None |
| inspection_started_at | None |
| instance_info | {} |
| instance_uuid | None |
| last_error | Failed to change power state to 'power on'. Error: 'int' object has no |
| | attribute 'find' |
| maintenance | True |
| maintenance_reason | During sync_power_state, max retries exceeded for node b7b49f65-04ff- |
| | 448c-a0b1-2bc89f64d862, node state None does not match expected state |
| | 'None'. Updating DB state to 'None' Switching node to maintenance mode. |
| | Error: 'int' object has no attribute 'find' |
| management_interface | |
| name | test-node |
| network_interface | |
| power_i...

[Lucas Alvares Gomes (lucasagomes)]

Hi Madhuri,

I noticed some things in your paste, the format of the "redfish_address" and "redfish_system_id" properties have changed from the patch that was in WIP before.

Now, "redfish_address" is only includes the scheme and authority part of the URL, and "redfish_system_d" includes whole path to the System resource.

So in your case, "redfish_address" would be "https://127.0.0.1:8443" and "redfish_system_id" would be "/redfish/v1/173"

See the docs: https://docs.openstack.org/developer/ironic/drivers/redfish.html#registering-a-node-with-the-redfish-driver

Can you make these changes and see if that works for you ?

[Lucas Alvares Gomes (lucasagomes)]

In any case, if changing it works for you we need to improve the property validation for the redfish driver so this bug is still valid.

[Lucas Alvares Gomes (lucasagomes)]

Ok, I can confirm that what triggers that error is the wrong properties being set so we need to improve validations. Here's the full error stack:

May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager [req-de8060e1-0c1e-4cca-8ff1-c62f655fc3f2 None None] During sync_power_state, could not get power state for node 728bcd9e-4250-485a-874e-0b8fd2aae289, attempt 2 of 3. Error: 'int' object has no attribute 'find'.
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager Traceback (most recent call last):
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/conductor/manager.py", line 2871, in do_sync_power_state
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager power_state = task.driver.power.get_power_state(task)
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/drivers/modules/redfish/power.py", line 87, in get_power_state
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager system = redfish_utils.get_system(task.node)
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/drivers/modules/redfish/utils.py", line 177, in get_system
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager return _get_system()
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 206, in call
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager return attempt.get(self._wrap_exception)
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager six.reraise(self.value[0], self.value[1], self.value[2])
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/drivers/modules/redfish/utils.py", line 159, in _get_system
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager return conn.get_system(system_id)
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager File "/usr/local/lib/python2.7/dist-packages/sushy/main.py", line 89, in get_system
May 05 10:27:37 ubuntu ironic-conductor[27852]: ERROR ironic.conductor.manager redfish_version=self.redfish_version)
May 05 10:...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/462898

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/462982

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462898
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=daf308322829b6fba163875581bb626a4d5b96b7
Submitter: Jenkins
Branch: master

commit daf308322829b6fba163875581bb626a4d5b96b7
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri May 5 11:01:27 2017 +0100

    Improve driver_info/redfish_system_id value validation

    This patch making sure that the value passed to
    driver_info/redfish_system_id is a string. The value of that property
    should be a canonical path to the System resource (in RedFish) that
    Ironic will manage.

    Change-Id: Ifb9e6e0cdb2b32370068fcee78edd136ff5ed7ce
    Closes-Bug: #1687658

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462982
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=fd19407952c9bd418990289806f2447543d8de71
Submitter: Jenkins
Branch: master

commit fd19407952c9bd418990289806f2447543d8de71
Author: Madhuri Kumari <email address hidden>
Date: Fri May 5 21:09:22 2017 +0530

    Improve driver_info/redfish_verify_ca value validation

    This patch adds a validation for redfish_verify_ca values
    'True' and 'False'. These string values are valid.

    Change-Id: I275eca6d3aa4ae25fdafcabbfd7725a8772c96fb
    Closes-Bug: #1687658

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 9.0.0 release.