Ironic conductor Hash ring reset needs to be independent of sync_local_state periodic task

Bug #1506657 reported by Meghal Gosalia
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
Zhenguo Niu

Bug Description

Currently, hash ring reset is done as part of sync_local_state periodic task [1].
But, for certain use cases, cloud admin will not be interested in running the sync_local_state periodic task and take over of active nodes by new conductor [2].
But, if sync_local_state is disabled, we will lose out on hash ring reset as well.
That will stop the dynamic redistribution of ironic nodes mapping to conductor [3] and will affect other periodic power sync jobs as well.

So, it would be good if hash ring rest is done as part of the mapping call itself and not tied to sync_local_state.

[1] - https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L1408
[2] - https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L1432
[3] - https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L1443

Changed in ironic:
status: New → Confirmed
importance: Undecided → Critical
tags: added: liberty-backport-potential
Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Talked about this in IRC some; we think the best solution is probably:

* Remove the reset from sync_local_state
* Keep track of when the ring was last refreshed
* In the code that checks if the ring is set, also check the age and rebuild if older than $config

Changed in ironic:
assignee: nobody → Zhenguo Niu (niu-zglinux)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/235806

Changed in ironic:
status: Confirmed → In Progress
Changed in ironic:
importance: Critical → High
Changed in ironic:
assignee: Zhenguo Niu (niu-zglinux) → John L. Villalovos (happycamp)
Changed in ironic:
assignee: John L. Villalovos (happycamp) → Zhenguo Niu (niu-zglinux)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/235806
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=fb1a67fe6bc919002e9a33aaf73a973fc48b7828
Submitter: Jenkins
Branch: master

commit fb1a67fe6bc919002e9a33aaf73a973fc48b7828
Author: Zhenguo Niu <email address hidden>
Date: Fri Oct 16 17:18:12 2015 +0800

    Move hash_ring refresh logic out of sync_local_state

    As sync_local_state periodic task can be disabled, which will lose
    out on hash ring refresh, so the refresh logic should be independent
    of it. this patch adds a updated_at to HashRingManager to track of
    when the ring was last refreshed, and when checks the ring is set,
    also check the age and rebuild if older than the new config option
    hash_ring_reset_interval which is default to 180s.

    Change-Id: Ie46dbf93b920543f99e11774a29878aaf27c3400
    Closes-Bug: #1506657

Changed in ironic:
status: In Progress → Fix Committed
Changed in ironic:
status: Fix Committed → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/ironic 4.3.0

This issue was fixed in the openstack/ironic 4.3.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.