Nova compute driver init happens before conductor is ready

Bug #1713095 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Dan Smith
Pike
Fix Committed
Medium
Lee Yarwood

Bug Description

In nova/service.py we poll for conductor readiness before we allow normal service startup behavior. The ironic driver does RPC to conductor in its _refresh_hash_ring() code, which may expect conductor be up before it's not. If so, we'll fail to start up because we called to conductor, waited a long time, and then timed out.

Tags: ironic
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/497966

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/497966
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=361287debf7c2cd5751517cedda8c960c5b5eb66
Submitter: Jenkins
Branch: master

commit 361287debf7c2cd5751517cedda8c960c5b5eb66
Author: Dan Smith <email address hidden>
Date: Fri Aug 25 08:48:49 2017 -0700

    Move hash ring initialization to init_host() for ironic

    The hash ring initialization code requires calling to conductor over
    RPC. Nova's service startup initializes the manager and the virt
    driver, then goes into a polling loop waiting for a conductor to
    respond to make sure it is ready. Finally, it calls init_host()
    once things are expected to be working. That is where we should
    do our initialization that requires conductor to be up.

    This patch moves the initial refresh into the Ironic driver's
    init_host() routine.

    Change-Id: If92f4b1df061f91ffffa7900532a978649efb933
    Closes-Bug: #1713095

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/502082

Matt Riedemann (mriedem)
tags: added: ironic
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/502082
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e53115da7fc10381c6f9cedcd909461e3c13c4ea
Submitter: Jenkins
Branch: stable/pike

commit e53115da7fc10381c6f9cedcd909461e3c13c4ea
Author: Dan Smith <email address hidden>
Date: Fri Aug 25 08:48:49 2017 -0700

    Move hash ring initialization to init_host() for ironic

    The hash ring initialization code requires calling to conductor over
    RPC. Nova's service startup initializes the manager and the virt
    driver, then goes into a polling loop waiting for a conductor to
    respond to make sure it is ready. Finally, it calls init_host()
    once things are expected to be working. That is where we should
    do our initialization that requires conductor to be up.

    This patch moves the initial refresh into the Ironic driver's
    init_host() routine.

    Change-Id: If92f4b1df061f91ffffa7900532a978649efb933
    Closes-Bug: #1713095
    (cherry picked from commit 361287debf7c2cd5751517cedda8c960c5b5eb66)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.1

This issue was fixed in the openstack/nova 16.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b1

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.