node_power_action should sync DB state with real state even if real state = requested state

[Dmitry Tantsur]

See the interesting part of the logs: http://paste.openstack.org/show/151894/

The scenario is:
1. We set maintenance=True
2. Power on the machine using API
3. ... do something with it ...
4. Power off the machine using API
5. Disable maintenance mode

One would expect the nodes to remain powered off. However pretty often sync power state loop seems to somehow kick in and power on the machine back.

UPDATE:
Setting force_power_state_during_sync=False in ironic.conf helps: http://paste.openstack.org/show/151899/

UPDATE2:
The problem manifested itself with VERY slow power driver (SSH'ing too seconds). Maybe kind of a race condition...

UPDATE3:
So that's how I see it:

1. Maintenance mode is set, power_state=off, real power state = off
2. Node is powered on, power_state = on, real power state = on
3. Discoverd ramdisk does something and reports back to discoverd
4. Discoverd instructs Ironic to power off the machine.
5. While it takes loooong (remember: very slow SSH) for Ironic to get power state, the ramdisk shutdowns itself
6. Now: get_power_state returns real power state = off and manager.utils.node_power_action exits
7. But! Power state in database was not updated and is still = on!
8. So after maintenance mode is disabled, Ironic powers on the machine.

SUGGESTION
During node_power_action _always_ make sure database is in sync with both what we expect and what we actually have (independent of force_power_state_during_sync value)

[aeva black (tenbrae)]

The previous message was the result of an incorrect bug number tag. I am reverting the status of this bug.

[Lucas Alvares Gomes (lucasagomes)]

I tried to reproduce it but I couldn't http://paste.openstack.org/show/157270/

I will take a better look into the code see if I can find something

[Dmitry Tantsur (divius)]

You need a slow power driver to reproduce it. Actually I think adding node.power_state = new_state after https://github.com/openstack/ironic/blob/master/ironic/conductor/utils.py#L88 should solve the problem.

In discoverd I do need power off command to be reliable.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/147494

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147494
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=d6787b2073eea16e29444581fa7b7791789be787
Submitter: Jenkins
Branch: master

commit d6787b2073eea16e29444581fa7b7791789be787
Author: Dmitry Tantsur <email address hidden>
Date: Thu Jan 15 13:53:44 2015 +0100

    Ensure we don't have stale power state in database after power action

    If the machine is shut down not via Ironic API (e.g. during maintenance)
    and then power off request is made, this request will succeed, but
    due to stale information in database the machine will be powered on
    as soon as power sync loop reaches it.

    Change-Id: Ia3ccaf1b3ab6399515c6eb6a98a0a85e3c2e6ddf
    Closes-Bug: #1403106