sync_power_state puts node in maintenance mode too fast

Bug #1304642 reported by Ruby Loo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
Medium
Yuriy Zveryanskyy

Bug Description

I created a node via 'ironic node-create -d fake_ipmitool', and in less than 5 minutes, the node was put in maintenance mode. I hadn't even finished setting it up yet :-(

If a node is new and hasn't been powered on yet, should we even be trying to sync the power state?

The logs show:

2014-04-08 19:49:41.130 4230 WARNING ironic.conductor.manager [-] During sync_power_state, could not get power state for node 35f26c4b-fccd-423b-9e37-7aa72f1c689b. Error: IPMI address not supplied to IPMI driver..
2014-04-08 19:50:41.184 4230 WARNING ironic.conductor.manager [-] During sync_power_state, could not get power state for node 35f26c4b-fccd-423b-9e37-7aa72f1c689b. Error: IPMI address not supplied to IPMI driver..
2014-04-08 19:51:41.186 4230 WARNING ironic.conductor.manager [-] During sync_power_state, could not get power state for node 35f26c4b-fccd-423b-9e37-7aa72f1c689b. Error: IPMI address not supplied to IPMI driver..
2014-04-08 19:51:41.199 4230 ERROR ironic.conductor.manager [-] During sync_power_state, max retries exceeded for node 35f26c4b-fccd-423b-9e37-7aa72f1c689b, node state None does not match expected state 'None'. Updating DB state to 'None' Switching node to maintenance mode.

Revision history for this message
Yuriy Zveryanskyy (yzveryanskyy) wrote :

Yes, I also think there is a problem. One possible solution: start power sync for node only if power state is not None (was set manually or after deploy)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/86529

Changed in ironic:
assignee: nobody → Yuriy Zveryanskyy (yzveryanskyy)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/86529
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=d648669f0c20b304adac803722da79ece172bc04
Submitter: Jenkins
Branch: master

commit d648669f0c20b304adac803722da79ece172bc04
Author: Yuriy Zveryanskyy <email address hidden>
Date: Thu Apr 10 11:32:49 2014 +0300

    Do not sync power state for new invalidated nodes

    sync_power_state periodic task puts node in maintenance mode after max
    sync retries. If node created and has unknown power state this
    periodic task set maintenance mode too fast (there is not yet IPMI
    credentials for example).
    This patch prevent sync_power_state for these nodes if validate fail.

    Closes-Bug: #1304642
    Change-Id: I19798028adce91b107643f5419999389dafdccf3

Changed in ironic:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ironic:
milestone: none → juno-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ironic:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.