tripleo

Cleanup Nova, Glance, NovaIronic services from undercloud while upgrading it

Bug #1959662 reported by Bogdan Dobrelya
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Rabi Mishra

Bug Description

As of Wallaby/OSP17 Nova/NovaIronic and Glance are no longer used on undercloud.
Keeping these services/containers around breaks upgrades

Bogdan Dobrelya (bogdando)
Changed in tripleo:
assignee: nobody → Bogdan Dobrelya (bogdando)
importance: Undecided → Critical
status: New → In Progress
tags: added: upgrade wallaby-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827260

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/827261

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

The issue looks deeper than that.
There is more of leftovers, like mistral_db_populate.

That means that we would have way more of init-like (one-time) containers to clean up for as well...

Perhaps a better solution would be for FFU to empty that container-startup-config location?
Or maybe make container manager smarter to detect old configs by managed_by instead of the enabled or not status of a service? See https://review.opendev.org/q/I595a998196b6de386a4ff345c39fe1e6a9ed62c6

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827268

Revision history for this message
Jose Luis Franco (jfrancoa) wrote :
Download full text (5.9 KiB)

When having a look at the logs, it seems that container_puppet_config.py considers this containers to be present also in the new upgraded set of services:

2022-02-01 08:36:59 | 2022-02-01 08:36:59.021577 | 52540022-e1f2-087c-6333-000000001340 | TASK | Ensure config hashes are up-to-date for container startup configs
2022-02-01 08:36:59 | 2022-02-01 08:36:59.458809 | | WARNING | Config change detected for mistral_db_populate, new hash: bb4e95c3b2f5c44780fdba1ca9d4acd2
2022-02-01 08:36:59 | 2022-02-01 08:36:59.459957 | | WARNING | Config change detected for nova_compute, new hash: dc6ad63723e5243699eda7448cd5bf07
2022-02-01 08:36:59 | 2022-02-01 08:36:59.460475 | | WARNING | Config change detected for nova_wait_for_compute_service, new hash: 0fad6766f7938e3133fe0c81949a72f1-0fad6766f7938e3133fe0c81949a72f1
2022-02-01 08:36:59 | 2022-02-01 08:36:59.460925 | | WARNING | Config change detected for haproxy, new hash: ba24f4565442b12bee9830c95e5bf6a8
2022-02-01 08:36:59 | 2022-02-01 08:36:59.461402 | | WARNING | Config change detected for memcached, new hash: 99ca38a19cba9d0be485aff39a84def2
2022-02-01 08:36:59 | 2022-02-01 08:36:59.461817 | | WARNING | Config change detected for rabbitmq, new hash: ad6edc0348579d9c2f519a4974fd1cfb
2022-02-01 08:36:59 | 2022-02-01 08:36:59.462293 | | WARNING | Config change detected for rabbitmq_bootstrap, new hash: ad6edc0348579d9c2f519a4974fd1cfb
2022-02-01 08:36:59 | 2022-02-01 08:36:59.462761 | | WARNING | Config change detected for mysql, new hash: 74e7556d5754c5278d65806b5d15b790
2022-02-01 08:36:59 | 2022-02-01 08:36:59.463186 | | WARNING | Config change detected for mysql_bootstrap, new hash: 74e7556d5754c5278d65806b5d15b790
2022-02-01 08:36:59 | 2022-02-01 08:36:59.463628 | | WARNING | Config change detected for mysql_upgrade_db, new hash: 74e7556d5754c5278d65806b5d15b790-74e7556d5754c5278d65806b5d15b790
2022-02-01 08:36:59 | 2022-02-01 08:36:59.464044 | | WARNING | Config change detected for ironic_db_sync, new hash: 75b79c11bdb2202b84b2c5f7b5c7a70b
2022-02-01 08:36:59 | 2022-02-01 08:36:59.464493 | | WARNING | Config change detected for ironic_inspector_db_sync, new hash: c87d856d29e5f1e942108a4facba4263
2022-02-01 08:36:59 | 2022-02-01 08:36:59.464956 | | WARNING | Config change detected for iscsid, new hash: 9e0c4e70b6156e39535070f2e225d54a
2022-02-01 08:36:59 | 2022-02-01 08:36:59.465399 | | WARNING | Config change detected for keystone, new hash: 8930d0d1a5cc77a59f9efaf5259a6dbf
2022-02-01 08:36:59 | 2022-02-01 08:36:59.465821 | | WARNING | Config change detected for keystone_cron, new hash: 8930d0d1a5cc77...

Read more...

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

@Jose It happens because its _find func matches all configs by the path matching a json pattern

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on python-tripleoclient (master)

Change abandoned by "Bogdan Dobrelya <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/827261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (master)

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/827257
Committed: https://opendev.org/openstack/python-tripleoclient/commit/a3c8ecf325b3b00c0f3d60398966b8b0d8e68ad0
Submitter: "Zuul (22348)"
Branch: master

commit a3c8ecf325b3b00c0f3d60398966b8b0d8e68ad0
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 1 10:50:06 2022 +0100

    Deprecate and unwire enable_nova UC option

    Since Wallaby Nova services aren't required in the Undercloud,
    therefore deprecate the enable_nova flag from the undercloud.conf.

    The undercloud-enable-nova.yaml env file will no longer be added
    to the deployment command, even if enable_nova is set in the config.

    Related-bug: #1959662
    Change-Id: I29e1b1b6ad1b5b3664f608a780512e1cbe3acacf
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827268
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/17004426aef236da2d79d299523c0a2e0abef03e
Submitter: "Zuul (22348)"
Branch: master

commit 17004426aef236da2d79d299523c0a2e0abef03e
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 1 12:37:09 2022 +0100

    Drop Nova and NovaJoin services from UC role data

    As of Wallaby, Nova service no longer in use on Undercloud.
    Keeping these in roles data waste container prepare cycles
    and the registry space for unused container images.

    Depends-On: https://review.opendev.org/827257
    Change-Id: I36367b58d43347ccf0fc18de4665a10891faef10
    Related-bug: #1959662
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/827647

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/827647
Committed: https://opendev.org/openstack/python-tripleoclient/commit/9fd8583f306413f422dd08a731e51e33ad03aff8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9fd8583f306413f422dd08a731e51e33ad03aff8
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 1 10:50:06 2022 +0100

    Deprecate and unwire enable_nova UC option

    Since Wallaby Nova services aren't required in the Undercloud,
    therefore deprecate the enable_nova flag from the undercloud.conf.

    The undercloud-enable-nova.yaml env file will no longer be added
    to the deployment command, even if enable_nova is set in the config.

    Related-bug: #1959662
    Change-Id: I29e1b1b6ad1b5b3664f608a780512e1cbe3acacf
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit a3c8ecf325b3b00c0f3d60398966b8b0d8e68ad0)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827806

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827742

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/828212

Bogdan Dobrelya (bogdando)
Changed in tripleo:
assignee: Bogdan Dobrelya (bogdando) → Rabi Mishra (rabi)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827968
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/5499a0760ad950ed77fd85db2b6cf1f8614e591f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5499a0760ad950ed77fd85db2b6cf1f8614e591f
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 1 12:37:09 2022 +0100

    Drop Nova and NovaJoin services from UC role data

    As of Wallaby, Nova service no longer in use on Undercloud.
    Keeping these in roles data waste container prepare cycles
    and the registry space for unused container images.

    Depends-On: https://review.opendev.org/827647
    Change-Id: I36367b58d43347ccf0fc18de4665a10891faef10
    Related-bug: #1959662
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 17004426aef236da2d79d299523c0a2e0abef03e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/wallaby)

Change abandoned by "Jose Luis Franco <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827742
Reason: Handled in https://review.opendev.org/c/openstack/tripleo-ansible/+/828212

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/828567

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/828567
Committed: https://opendev.org/openstack/tripleo-ansible/commit/36b2175afb1243efa9a5f40aa0f6eee2c5c8e934
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 36b2175afb1243efa9a5f40aa0f6eee2c5c8e934
Author: rabi <email address hidden>
Date: Tue Feb 8 08:38:24 2022 +0530

    Remove old per step startup configs

    There may be cases where certain steps may not exist after
    removal of services during update/upgrade. As we recreate
    the startup configs per step every time, let's clean all.
    If we need to archive old configs we can backup docker_config.yaml
    which can be done in a separate THT patch.

    Change-Id: I200865bfd03aa63a9fe07b517bf60151e495455f
    Closes-bug: #1959662

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by "Jose Luis Franco <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827806
Reason: Fixed by https://review.opendev.org/c/openstack/tripleo-ansible/+/828212

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/828212
Committed: https://opendev.org/openstack/tripleo-ansible/commit/487f127b70d846db9fde0b6e90f463c26af68bd1
Submitter: "Zuul (22348)"
Branch: master

commit 487f127b70d846db9fde0b6e90f463c26af68bd1
Author: rabi <email address hidden>
Date: Tue Feb 8 08:38:24 2022 +0530

    Remove old per step startup configs

    There may be cases where certain steps may not exist after
    removal of services during update/upgrade. As we recreate
    the startup configs per step every time, let's clean all.
    If we need to archive old configs we can backup docker_config.yaml
    which can be done in a separate THT patch.

    Change-Id: I200865bfd03aa63a9fe07b517bf60151e495455f
    Closes-bug: #1959662

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827260
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/219817528f9665e21df02686f2ed2da0bd2241b5
Submitter: "Zuul (22348)"
Branch: master

commit 219817528f9665e21df02686f2ed2da0bd2241b5
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 1 11:28:15 2022 +0100

    Remove Nova from undercloud during upgrades

    Note that one-off init containers should be removed as well.

    Related-bug: #1959662
    Change-Id: I9380cc4c62acc6860f7a1e5869131fa3406e2d98
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/833736

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/833737

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/833738

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/833736
Committed: https://opendev.org/openstack/tripleo-ansible/commit/f48f29034a803f35079f9fd0143509306a792c49
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit f48f29034a803f35079f9fd0143509306a792c49
Author: rabi <email address hidden>
Date: Tue Feb 8 08:38:24 2022 +0530

    Remove old per step startup configs

    There may be cases where certain steps may not exist after
    removal of services during update/upgrade. As we recreate
    the startup configs per step every time, let's clean all.
    If we need to archive old configs we can backup docker_config.yaml
    which can be done in a separate THT patch.

    Change-Id: I200865bfd03aa63a9fe07b517bf60151e495455f
    Closes-bug: #1959662
    (cherry picked from commit 36b2175afb1243efa9a5f40aa0f6eee2c5c8e934)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/833737
Committed: https://opendev.org/openstack/tripleo-ansible/commit/7bd05fa05582a96088afa51e76ad3afd8e80679f
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 7bd05fa05582a96088afa51e76ad3afd8e80679f
Author: rabi <email address hidden>
Date: Tue Feb 8 08:38:24 2022 +0530

    Remove old per step startup configs

    There may be cases where certain steps may not exist after
    removal of services during update/upgrade. As we recreate
    the startup configs per step every time, let's clean all.
    If we need to archive old configs we can backup docker_config.yaml
    which can be done in a separate THT patch.

    Change-Id: I200865bfd03aa63a9fe07b517bf60151e495455f
    Closes-bug: #1959662
    (cherry picked from commit 36b2175afb1243efa9a5f40aa0f6eee2c5c8e934)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/train)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/833738
Committed: https://opendev.org/openstack/tripleo-ansible/commit/fa2967f710cb010104000de76bdfd2645f6c4735
Submitter: "Zuul (22348)"
Branch: stable/train

commit fa2967f710cb010104000de76bdfd2645f6c4735
Author: rabi <email address hidden>
Date: Tue Feb 8 08:38:24 2022 +0530

    Remove old per step startup configs

    There may be cases where certain steps may not exist after
    removal of services during update/upgrade. As we recreate
    the startup configs per step every time, let's clean all.
    If we need to archive old configs we can backup docker_config.yaml
    which can be done in a separate THT patch.

    Change-Id: I200865bfd03aa63a9fe07b517bf60151e495455f
    Closes-bug: #1959662
    (cherry picked from commit 36b2175afb1243efa9a5f40aa0f6eee2c5c8e934)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 4.2.0

This issue was fixed in the openstack/tripleo-ansible 4.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible ussuri-eol

This issue was fixed in the openstack/tripleo-ansible ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 2.6.0

This issue was fixed in the openstack/tripleo-ansible 2.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847154

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847154
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/2f21e680690487b6353a6f6002f142298aba8a15
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2f21e680690487b6353a6f6002f142298aba8a15
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 1 11:28:15 2022 +0100

    Remove Nova from undercloud during upgrades

    Note that one-off init containers should be removed as well.

    Related-bug: #1959662
    Change-Id: I9380cc4c62acc6860f7a1e5869131fa3406e2d98
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 219817528f9665e21df02686f2ed2da0bd2241b5)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible train-eol

This issue was fixed in the openstack/tripleo-ansible train-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.