Get/Set power state fails

[Christopher Dearborn]

The periodic job to retrieve node power state fails on specific nodes resulting in the nodes being moved into maintenance mode and an unknown power state.

or

A command to set the power state on specific nodes fails during the initial query to get the power state, resulting in the set power state command not being executed against those nodes and so the nodes fail to change power state.

Investigation:
Investigation has shown that it is possible for iDRACs to get into a state where the "Name" element is completely missing from the DCIM_ComputerSystem table. This results in the following query issued from get_power_state resulting in 0 records, and the resulting attempt to retrieve the power state fails and an exception is thrown:
select EnabledState from DCIM_ComputerSystem where Name="srv:system"

Resetting the iDRAC causes the "Name" element to be reported once again on a query of the DCIM_ComputerSystem table, which causes get_power_state to now succeed, where before it failed.

This has been seen on multiple servers that are running the most recent firmware (lifecycle controller: 2.41.40.40, bios: 2.5.4).

Note that the DCIM_ComputerSystem table appears to always contain only 1 record, making selecting on Name unnecessary. I tried removing the select on Name from the get_power_state select and it worked both on nodes that were in this state and on nodes that were not.

It should be noted that the command to set the power state uses "'Name': 'srv:system'" in the selector. Experimentation showed that this command works both on iDRACs that are in the state where no "Name" element is returned when querying DCIM_ComputerSystem, and iDRACs that are not in that state. I also tried changing this selector to not specify Name in the selector, but the command fails in this case.

I have contacted the iDRAC team and they agree that this is a firmware bug. It will probably be a considerable period of time before it is fixed in the firmware though.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/437132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/437132
Committed: https://git.openstack.org/cgit/openstack/python-dracclient/commit/?id=f49efaa1bfd20eb7d94366f0665df01d9b0d10b4
Submitter: Jenkins
Branch: master

commit f49efaa1bfd20eb7d94366f0665df01d9b0d10b4
Author: Christopher Dearborn <email address hidden>
Date: Wed Feb 22 15:19:41 2017 -0500

    Get/Set power state fails

    It's possible for iDRACs to get into a state where the Name element is
    not returned when querying DCIM_ComputerSystem. This results in the
    get_power_state command failing against nodes with iDRACs in this
    state.

    This patch removes selecting on the Name element from get_power_state.
    This makes get_power_state work against both nodes with iDRACs in this
    state and iDRACs not in this state.

    Change-Id: Ide2d69b695376c777e5c4ee847dd0e481474a09f
    Closes-Bug: #1667088