kolla-ansible

Nova once-per-cell tasks not fatal for all hosts

Bug #1948694 reported by Mark Goddard
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Unassigned

Bug Description

We run some nova tasks once per cell, using a condition to match a single host in the cell. In other similar tasks, we use run_once, which will fail all hosts if the task fails. Typically these tasks are critical, and that is desirable. However, with the approach used in nova-cell to support multiple cells, if a once-per-cell task fails, then other hosts will continue to execute, which could lead to unexpected results.

Examples of once-per-cell tasks:

* Creating Nova cell database
* Run Nova cell online database migrations

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/815335

Changed in kolla-ansible:
status: New → In Progress
Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/815335
Committed: https://opendev.org/openstack/kolla-ansible/commit/832989d0a6ab9ba1380d559e0e532c17db82c2c6
Submitter: "Zuul (22348)"
Branch: master

commit 832989d0a6ab9ba1380d559e0e532c17db82c2c6
Author: Mark Goddard <email address hidden>
Date: Mon Oct 25 16:40:20 2021 +0100

    nova: use any_errors_fatal for once-per-cell tasks

    We run some nova tasks once per cell, using a condition to match a
    single host in the cell. In other similar tasks, we use run_once, which
    will fail all hosts if the task fails. Typically these tasks are
    critical, and that is desirable. However, with the approach used in
    nova-cell to support multiple cells, if a once-per-cell task fails, then
    other hosts will continue to execute, which could lead to unexpected
    results.

    This change adds any_errors_fatal to the plays or blocks that run these
    tasks.

    Closes-Bug: #1948694

    Change-Id: I2a5871ccd4e8198171ef3239ce95f475f3e4b051

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839172

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839173

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839174

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839175

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/839176

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839176
Committed: https://opendev.org/openstack/kolla-ansible/commit/52e36675169d095ce028129ecdf95e0669c1434c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 52e36675169d095ce028129ecdf95e0669c1434c
Author: Mark Goddard <email address hidden>
Date: Mon Oct 25 16:40:20 2021 +0100

    nova: use any_errors_fatal for once-per-cell tasks

    We run some nova tasks once per cell, using a condition to match a
    single host in the cell. In other similar tasks, we use run_once, which
    will fail all hosts if the task fails. Typically these tasks are
    critical, and that is desirable. However, with the approach used in
    nova-cell to support multiple cells, if a once-per-cell task fails, then
    other hosts will continue to execute, which could lead to unexpected
    results.

    This change adds any_errors_fatal to the plays or blocks that run these
    tasks.

    Closes-Bug: #1948694

    Change-Id: I2a5871ccd4e8198171ef3239ce95f475f3e4b051
    (cherry picked from commit 832989d0a6ab9ba1380d559e0e532c17db82c2c6)

tags: added: in-stable-ussuri
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839175
Committed: https://opendev.org/openstack/kolla-ansible/commit/3c10ce0d8623d5e900154d18c12966618340b50a
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 3c10ce0d8623d5e900154d18c12966618340b50a
Author: Mark Goddard <email address hidden>
Date: Mon Oct 25 16:40:20 2021 +0100

    nova: use any_errors_fatal for once-per-cell tasks

    We run some nova tasks once per cell, using a condition to match a
    single host in the cell. In other similar tasks, we use run_once, which
    will fail all hosts if the task fails. Typically these tasks are
    critical, and that is desirable. However, with the approach used in
    nova-cell to support multiple cells, if a once-per-cell task fails, then
    other hosts will continue to execute, which could lead to unexpected
    results.

    This change adds any_errors_fatal to the plays or blocks that run these
    tasks.

    Closes-Bug: #1948694

    Change-Id: I2a5871ccd4e8198171ef3239ce95f475f3e4b051
    (cherry picked from commit 832989d0a6ab9ba1380d559e0e532c17db82c2c6)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839174
Committed: https://opendev.org/openstack/kolla-ansible/commit/fdb52f71acb462a4ade2e82a642c4bddd0c6f1a8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit fdb52f71acb462a4ade2e82a642c4bddd0c6f1a8
Author: Mark Goddard <email address hidden>
Date: Mon Oct 25 16:40:20 2021 +0100

    nova: use any_errors_fatal for once-per-cell tasks

    We run some nova tasks once per cell, using a condition to match a
    single host in the cell. In other similar tasks, we use run_once, which
    will fail all hosts if the task fails. Typically these tasks are
    critical, and that is desirable. However, with the approach used in
    nova-cell to support multiple cells, if a once-per-cell task fails, then
    other hosts will continue to execute, which could lead to unexpected
    results.

    This change adds any_errors_fatal to the plays or blocks that run these
    tasks.

    Closes-Bug: #1948694

    Change-Id: I2a5871ccd4e8198171ef3239ce95f475f3e4b051
    (cherry picked from commit 832989d0a6ab9ba1380d559e0e532c17db82c2c6)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839173
Committed: https://opendev.org/openstack/kolla-ansible/commit/d1b7814c76e309791353ad4a69ae6dfd14c75723
Submitter: "Zuul (22348)"
Branch: stable/xena

commit d1b7814c76e309791353ad4a69ae6dfd14c75723
Author: Mark Goddard <email address hidden>
Date: Mon Oct 25 16:40:20 2021 +0100

    nova: use any_errors_fatal for once-per-cell tasks

    We run some nova tasks once per cell, using a condition to match a
    single host in the cell. In other similar tasks, we use run_once, which
    will fail all hosts if the task fails. Typically these tasks are
    critical, and that is desirable. However, with the approach used in
    nova-cell to support multiple cells, if a once-per-cell task fails, then
    other hosts will continue to execute, which could lead to unexpected
    results.

    This change adds any_errors_fatal to the plays or blocks that run these
    tasks.

    Closes-Bug: #1948694

    Change-Id: I2a5871ccd4e8198171ef3239ce95f475f3e4b051
    (cherry picked from commit 832989d0a6ab9ba1380d559e0e532c17db82c2c6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/839172
Committed: https://opendev.org/openstack/kolla-ansible/commit/b141d82f043719f5b7a366e6e4fac08d3becb267
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit b141d82f043719f5b7a366e6e4fac08d3becb267
Author: Mark Goddard <email address hidden>
Date: Mon Oct 25 16:40:20 2021 +0100

    nova: use any_errors_fatal for once-per-cell tasks

    We run some nova tasks once per cell, using a condition to match a
    single host in the cell. In other similar tasks, we use run_once, which
    will fail all hosts if the task fails. Typically these tasks are
    critical, and that is desirable. However, with the approach used in
    nova-cell to support multiple cells, if a once-per-cell task fails, then
    other hosts will continue to execute, which could lead to unexpected
    results.

    This change adds any_errors_fatal to the plays or blocks that run these
    tasks.

    Closes-Bug: #1948694

    Change-Id: I2a5871ccd4e8198171ef3239ce95f475f3e4b051
    (cherry picked from commit 832989d0a6ab9ba1380d559e0e532c17db82c2c6)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 14.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 12.4.0

This issue was fixed in the openstack/kolla-ansible 12.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 13.1.0

This issue was fixed in the openstack/kolla-ansible 13.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 15.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 15.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible ussuri-eol

This issue was fixed in the openstack/kolla-ansible ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible victoria-eol

This issue was fixed in the openstack/kolla-ansible victoria-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.