node_power_action should sync DB state with real state even if real state = requested state
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ironic |
Fix Released
|
High
|
Dmitry Tantsur |
Bug Description
See the interesting part of the logs: http://
The scenario is:
1. We set maintenance=True
2. Power on the machine using API
3. ... do something with it ...
4. Power off the machine using API
5. Disable maintenance mode
One would expect the nodes to remain powered off. However pretty often sync power state loop seems to somehow kick in and power on the machine back.
UPDATE:
Setting force_power_
UPDATE2:
The problem manifested itself with VERY slow power driver (SSH'ing too seconds). Maybe kind of a race condition...
UPDATE3:
So that's how I see it:
1. Maintenance mode is set, power_state=off, real power state = off
2. Node is powered on, power_state = on, real power state = on
3. Discoverd ramdisk does something and reports back to discoverd
4. Discoverd instructs Ironic to power off the machine.
5. While it takes loooong (remember: very slow SSH) for Ironic to get power state, the ramdisk shutdowns itself
6. Now: get_power_state returns real power state = off and manager.
7. But! Power state in database was not updated and is still = on!
8. So after maintenance mode is disabled, Ironic powers on the machine.
SUGGESTION
During node_power_action _always_ make sure database is in sync with both what we expect and what we actually have (independent of force_power_
description: | updated |
Changed in ironic: | |
assignee: | nobody → Devananda van der Veen (devananda) |
status: | New → In Progress |
description: | updated |
description: | updated |
summary: |
- force_power_state_during_sync can force an incorrect power state after - leaving maintenance mode + node_power_action should sync real state with DB state even if real + state = requested state |
summary: |
- node_power_action should sync real state with DB state even if real + node_power_action should sync DB state with real state even if real state = requested state |
Changed in ironic: | |
status: | New → Confirmed |
importance: | Undecided → High |
tags: | added: low-hanging-fruit |
Changed in ironic: | |
status: | Confirmed → In Progress |
assignee: | nobody → Dmitry "Divius" Tantsur (divius) |
Changed in ironic: | |
milestone: | none → kilo-2 |
status: | Fix Committed → Fix Released |
Changed in ironic: | |
milestone: | kilo-2 → 2015.1.0 |
The previous message was the result of an incorrect bug number tag. I am reverting the status of this bug.