cell_v2 discovery does not run during scale out when controllers are blacklisted

Bug #1786961 reported by Oliver Walsh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Rajesh Tailor

Bug Description

When scaling out compute nodes is it possible to blacklist the controller nodes to avoid pushing any changes to controllers. If this approach is used the new compute nodes will not have cell mappings as we currently run cell_v2 discovery on the controller nodes.

More info in https://bugzilla.redhat.com/show_bug.cgi?id=1562082

Oliver Walsh (owalsh)
Changed in tripleo:
status: New → In Progress
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Oliver Walsh (owalsh)
tags: added: ocata-backport-potential pike-backport-potential rocky-backport-potential
tags: removed: ocata-backport-potential
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
assignee: Oliver Walsh (owalsh) → Rajesh Tailor (ratailor)
Changed in tripleo:
milestone: stein-2 → stein-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/576481
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e0e885b8ca3332e0815c537a32c564cac81f7f7e
Submitter: Zuul
Branch: master

commit e0e885b8ca3332e0815c537a32c564cac81f7f7e
Author: Oliver Walsh <email address hidden>
Date: Tue Jun 19 11:56:16 2018 +0100

    Move cellv2 discovery from control plane services to compute services

    If compute nodes are deployed without deploying/updating the controllers then
    the computes will not have cellv2 mappings as this is run in the controller
    deploy steps (nova-api).
    This can happen if the controller nodes are blacklisted during a compute scale
    out. It's also likely to be an issue going forward if the deployment is staged
    (e.g split control plane).

    This change moves the cell_v2 discovery logic to the nova-compute/nova-ironic
    deploy step.

    Closes-bug: 1786961
    Change-Id: I12a02f636f31985bc1b71bff5b744d346286a95f

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/639920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/639920
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d7704bbf56cc155884e233a816422aa6e721cda3
Submitter: Zuul
Branch: stable/rocky

commit d7704bbf56cc155884e233a816422aa6e721cda3
Author: Oliver Walsh <email address hidden>
Date: Tue Jun 19 11:56:16 2018 +0100

    Move cellv2 discovery from control plane services to compute services

    If compute nodes are deployed without deploying/updating the controllers then
    the computes will not have cellv2 mappings as this is run in the controller
    deploy steps (nova-api).
    This can happen if the controller nodes are blacklisted during a compute scale
    out. It's also likely to be an issue going forward if the deployment is staged
    (e.g split control plane).

    This change moves the cell_v2 discovery logic to the nova-compute/nova-ironic
    deploy step.

    Conflicts:
           common/services.yaml
           docker/services/nova-api.yaml

    Closes-bug: 1786961
    Change-Id: I12a02f636f31985bc1b71bff5b744d346286a95f
    (cherry picked from commit e0e885b8ca3332e0815c537a32c564cac81f7f7e)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/641968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.4.0

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/641968
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8d77ea1691f60eb7229404f3c33c0cd833b3756b
Submitter: Zuul
Branch: stable/queens

commit 8d77ea1691f60eb7229404f3c33c0cd833b3756b
Author: Oliver Walsh <email address hidden>
Date: Tue Jun 19 11:56:16 2018 +0100

    Move cellv2 discovery from control plane services to compute services

    If compute nodes are deployed without deploying/updating the controllers then
    the computes will not have cellv2 mappings as this is run in the controller
    deploy steps (nova-api).
    This can happen if the controller nodes are blacklisted during a compute scale
    out. It's also likely to be an issue going forward if the deployment is staged
    (e.g split control plane).

    This change moves the cell_v2 discovery logic to the nova-compute/nova-ironic
    deploy step.

    Conflicts:
           common/services.yaml
           docker/services/nova-api.yaml

    Closes-bug: 1786961
    Change-Id: I12a02f636f31985bc1b71bff5b744d346286a95f
    (cherry picked from commit e0e885b8ca3332e0815c537a32c564cac81f7f7e)
    (cherry picked from commit d7704bbf56cc155884e233a816422aa6e721cda3)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.4.0

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.4.0

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/762579
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/629485dde5d6b96d38688f362362fac45392ece9
Submitter: Zuul
Branch: master

commit 629485dde5d6b96d38688f362362fac45392ece9
Author: Oliver Walsh <email address hidden>
Date: Thu Nov 12 20:38:26 2020 +0000

    Move cell_v2 discovery off compute hosts

    In I12a02f636f31985bc1b71bff5b744d346286a95f cell_v2 discovery was
    originally moved from the nova-api container to the
    nova-compute|nova-ironic containers in order to run cell
    discovery during a scale up where the controllers are omitted
    (e.g to exclude the controllers from a maintenance window).

    This requires api database credentials on the compute node, which is
    forbidden, so it must move back to a nova-api host as a pre-requisite
    for removing these credentials in a follow-up patch.

    Scale-up while omitting the controllers will no longer work out of the
    box. Either a manual cell_v2 discovery can be run after scale up, or an
    additional node can be deployed using the NovaManager tripleo role.

    Related-bug: #1786961
    Related-bug: #1871482
    Change-Id: I47b95ad46e2d4e5b1f370a2f840826e87da2d703

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.