Nova compute service registration failure fails the API querying host

Bug #1940119 reported by Mark Goddard
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Unassigned

Bug Description

If any nova compute service fails to register itself, Kolla Ansible will fail the first host in the inventory from that cell - the one that queries the nova API in the task 'Waiting for nova-compute services to register themselves'. Other hosts continue, often leading to further errors later on.

Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)
Changed in kolla-ansible:
status: New → In Progress
Mark Goddard (mgoddard)
description: updated
summary: - Singular Nova compute service registration failure fails all hosts
+ Nova compute service registration failure fails the querying host
summary: - Nova compute service registration failure fails the querying host
+ Nova compute service registration failure fails the API querying host
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/805599
Committed: https://opendev.org/openstack/kolla-ansible/commit/f1d3ff11d0e43b7e70fe9c36709975d96dfa86e8
Submitter: "Zuul (22348)"
Branch: master

commit f1d3ff11d0e43b7e70fe9c36709975d96dfa86e8
Author: Mark Goddard <email address hidden>
Date: Mon Aug 16 17:10:18 2021 +0100

    nova: improve compute service registration failure handling

    If any nova compute service fails to register itself, Kolla Ansible will
    fail the host that queries the Nova API. This is the first compute host
    in the inventory, and fails in the task:

        Waiting for nova-compute services to register themselves

    Other hosts continue, often leading to further errors later on. Clearly
    this is not idea.

    This change modifies the behaviour to query the compute service list
    until all expected hosts are present, but does not fail the querying
    host if they are not. A new task is added that executes for all hosts,
    and fails only those hosts that have not registered successfully.

    Alternatively, to fail all hosts in a cell when any compute service
    fails to register, set nova_compute_registration_fatal to true.

    Change-Id: I12c1928cf1f1fb9e28f1741e7fe4968004ea1816
    Closes-Bug: #1940119

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839177

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839178

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839179

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839331

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839177
Committed: https://opendev.org/openstack/kolla-ansible/commit/a899865655f8559e671cdf926455cfb35a0e69ca
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit a899865655f8559e671cdf926455cfb35a0e69ca
Author: Mark Goddard <email address hidden>
Date: Mon Aug 16 17:10:18 2021 +0100

    nova: improve compute service registration failure handling

    If any nova compute service fails to register itself, Kolla Ansible will
    fail the host that queries the Nova API. This is the first compute host
    in the inventory, and fails in the task:

        Waiting for nova-compute services to register themselves

    Other hosts continue, often leading to further errors later on. Clearly
    this is not idea.

    This change modifies the behaviour to query the compute service list
    until all expected hosts are present, but does not fail the querying
    host if they are not. A new task is added that executes for all hosts,
    and fails only those hosts that have not registered successfully.

    Alternatively, to fail all hosts in a cell when any compute service
    fails to register, set nova_compute_registration_fatal to true.

    Change-Id: I12c1928cf1f1fb9e28f1741e7fe4968004ea1816
    Closes-Bug: #1940119
    (cherry picked from commit f1d3ff11d0e43b7e70fe9c36709975d96dfa86e8)

tags: added: in-stable-yoga
tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839331
Committed: https://opendev.org/openstack/kolla-ansible/commit/14ce3053029965c3ba76af982eeeadb37f06ea2b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 14ce3053029965c3ba76af982eeeadb37f06ea2b
Author: Mark Goddard <email address hidden>
Date: Mon Aug 16 17:10:18 2021 +0100

    nova: improve compute service registration failure handling

    If any nova compute service fails to register itself, Kolla Ansible will
    fail the host that queries the Nova API. This is the first compute host
    in the inventory, and fails in the task:

        Waiting for nova-compute services to register themselves

    Other hosts continue, often leading to further errors later on. Clearly
    this is not idea.

    This change modifies the behaviour to query the compute service list
    until all expected hosts are present, but does not fail the querying
    host if they are not. A new task is added that executes for all hosts,
    and fails only those hosts that have not registered successfully.

    Alternatively, to fail all hosts in a cell when any compute service
    fails to register, set nova_compute_registration_fatal to true.

    Change-Id: I12c1928cf1f1fb9e28f1741e7fe4968004ea1816
    Closes-Bug: #1940119
    (cherry picked from commit f1d3ff11d0e43b7e70fe9c36709975d96dfa86e8)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839178
Committed: https://opendev.org/openstack/kolla-ansible/commit/219c39500960dd8c771d9085f8b61c2124ecec79
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 219c39500960dd8c771d9085f8b61c2124ecec79
Author: Mark Goddard <email address hidden>
Date: Mon Aug 16 17:10:18 2021 +0100

    nova: improve compute service registration failure handling

    If any nova compute service fails to register itself, Kolla Ansible will
    fail the host that queries the Nova API. This is the first compute host
    in the inventory, and fails in the task:

        Waiting for nova-compute services to register themselves

    Other hosts continue, often leading to further errors later on. Clearly
    this is not idea.

    This change modifies the behaviour to query the compute service list
    until all expected hosts are present, but does not fail the querying
    host if they are not. A new task is added that executes for all hosts,
    and fails only those hosts that have not registered successfully.

    Alternatively, to fail all hosts in a cell when any compute service
    fails to register, set nova_compute_registration_fatal to true.

    Change-Id: I12c1928cf1f1fb9e28f1741e7fe4968004ea1816
    Closes-Bug: #1940119
    (cherry picked from commit f1d3ff11d0e43b7e70fe9c36709975d96dfa86e8)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839179
Committed: https://opendev.org/openstack/kolla-ansible/commit/03d29f414af972019173f8bd4b684cb976c92bd5
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 03d29f414af972019173f8bd4b684cb976c92bd5
Author: Mark Goddard <email address hidden>
Date: Mon Aug 16 17:10:18 2021 +0100

    nova: improve compute service registration failure handling

    If any nova compute service fails to register itself, Kolla Ansible will
    fail the host that queries the Nova API. This is the first compute host
    in the inventory, and fails in the task:

        Waiting for nova-compute services to register themselves

    Other hosts continue, often leading to further errors later on. Clearly
    this is not idea.

    This change modifies the behaviour to query the compute service list
    until all expected hosts are present, but does not fail the querying
    host if they are not. A new task is added that executes for all hosts,
    and fails only those hosts that have not registered successfully.

    Alternatively, to fail all hosts in a cell when any compute service
    fails to register, set nova_compute_registration_fatal to true.

    Change-Id: I12c1928cf1f1fb9e28f1741e7fe4968004ea1816
    Closes-Bug: #1940119
    (cherry picked from commit f1d3ff11d0e43b7e70fe9c36709975d96dfa86e8)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839180
Committed: https://opendev.org/openstack/kolla-ansible/commit/d499f573a72fa339a276a012967feb2b94fca56b
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit d499f573a72fa339a276a012967feb2b94fca56b
Author: Mark Goddard <email address hidden>
Date: Mon Aug 16 17:10:18 2021 +0100

    nova: improve compute service registration failure handling

    If any nova compute service fails to register itself, Kolla Ansible will
    fail the host that queries the Nova API. This is the first compute host
    in the inventory, and fails in the task:

        Waiting for nova-compute services to register themselves

    Other hosts continue, often leading to further errors later on. Clearly
    this is not idea.

    This change modifies the behaviour to query the compute service list
    until all expected hosts are present, but does not fail the querying
    host if they are not. A new task is added that executes for all hosts,
    and fails only those hosts that have not registered successfully.

    Alternatively, to fail all hosts in a cell when any compute service
    fails to register, set nova_compute_registration_fatal to true.

    Change-Id: I12c1928cf1f1fb9e28f1741e7fe4968004ea1816
    Closes-Bug: #1940119
    (cherry picked from commit f1d3ff11d0e43b7e70fe9c36709975d96dfa86e8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 14.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 12.4.0

This issue was fixed in the openstack/kolla-ansible 12.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 13.1.0

This issue was fixed in the openstack/kolla-ansible 13.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 15.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 15.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible ussuri-eol

This issue was fixed in the openstack/kolla-ansible ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible victoria-eol

This issue was fixed in the openstack/kolla-ansible victoria-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.