Hash ring is stale after new conductor added

Bug #1378570 reported by Jim Rollenhagen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
Critical
Jim Rollenhagen

Bug Description

A previous commit ( https://review.openstack.org/#/c/120951/ ) introduced a change where the hash ring is cached, and reset on every request. However, it's not actually reset, and adding a conductor does not actually cause that conductor to be added to the local copy of the hash ring on the API services.

To reproduce:

* Spin up an API service, make some request that should go to a conductor (e.g. node-update). Observe expected behavior of "conductor not found that supports driver" or similar.
* Start a conductor service loaded with the driver for that node.
* Make the same node-update request. Observe erroneous behavior of "conductor not found that supports driver" or similar.
* Restart the API service. Make the same node-update request. Observe expected success.

Changed in ironic:
assignee: nobody → Jim Rollenhagen (jim-rollenhagen)
status: New → In Progress
Changed in ironic:
importance: Undecided → Critical
tags: added: juno-rc-potential
Revision history for this message
aeva black (tenbrae) wrote :

Confirmed.

This one is my fault - amongst the various patches to fix the hash ring, I forgot to add a reset() call. Thanks for catching this, Jim!

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/126751
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=d7cc83129966109556df8669aa970ff39fd156ed
Submitter: Jenkins
Branch: master

commit d7cc83129966109556df8669aa970ff39fd156ed
Author: Jim Rollenhagen <email address hidden>
Date: Tue Oct 7 16:24:31 2014 -0700

    Clear hash ring cache in get_topic_for*

    This causes the hash ring to be reset on each API request, avoiding
    an issue where a new conductor never joins the ring until the API
    service is restarted.

    Closes-Bug: 1378570

    Change-Id: Ic57370163f2f833efab662837036298f29f4ca90

Changed in ironic:
status: In Progress → Fix Committed
Revision history for this message
Dmitry Tantsur (divius) wrote :

Are we backporting it for J?

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Yes, we are.

aeva black (tenbrae)
Changed in ironic:
milestone: none → juno-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (proposed/juno)

Fix proposed to branch: proposed/juno
Review: https://review.openstack.org/126984

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (proposed/juno)

Reviewed: https://review.openstack.org/126984
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=3807e9447634c5ce6d9f65cac48ddc9bc0f9d295
Submitter: Jenkins
Branch: proposed/juno

commit 3807e9447634c5ce6d9f65cac48ddc9bc0f9d295
Author: Jim Rollenhagen <email address hidden>
Date: Tue Oct 7 16:24:31 2014 -0700

    Clear hash ring cache in get_topic_for*

    This causes the hash ring to be reset on each API request, avoiding
    an issue where a new conductor never joins the ring until the API
    service is restarted.

    Closes-Bug: 1378570

    Change-Id: Ic57370163f2f833efab662837036298f29f4ca90

Changed in ironic:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ironic:
milestone: juno-rc2 → 2014.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/128965

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.