tripleo

openstack overcloud delete errors out in master

Bug #1938618 reported by Michele Baldessari on 2021-07-31

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Medium	Sandeep Yadav

Bug Description

With ephemeral heat on my undercloud:
(undercloud) [stack@undercloud-0 ~]$ sudo podman ps |grep heat
05425f3a9d02 localhost/tripleo/openstack-heat-engine:ephemeral 8 minutes ago Up 8 minutes ago 192.168.4.2:8006->8006/tcp ephemeral-heat-engine
44ac80f01845 localhost/tripleo/openstack-heat-api:ephemeral 8 minutes ago Up 8 minutes ago 192.168.4.2:8006->8006/tcp ephemeral-heat-api

openstack overcloud delete --yes overcloud fails with:
(undercloud) [stack@undercloud-0 ~]$ openstack overcloud delete --yes overcloud

PLAY [Overcloud Delete] ********************************************************
2021-07-31 14:48:05.792090 | 525400d7-b628-d15b-cf41-000000000015 | TASK | Check for required inputs
2021-07-31 14:48:05.828039 | 525400d7-b628-d15b-cf41-000000000015 | SKIPPED | Check for required inputs | localhost | item=stack_name
2021-07-31 14:48:05.829412 | 525400d7-b628-d15b-cf41-000000000015 | TIMING | Check for required inputs | localhost | 0:00:02.803428 | 0.04s
2021-07-31 14:48:05.834357 | 525400d7-b628-d15b-cf41-000000000019 | TASK | Delete service vips used
2021-07-31 14:48:07.486751 | 525400d7-b628-d15b-cf41-000000000019 | CHANGED | Delete service vips used | localhost
2021-07-31 14:48:07.488125 | 525400d7-b628-d15b-cf41-000000000019 | TIMING | Delete service vips used | localhost | 0:00:04.462121 | 1.65s
2021-07-31 14:48:07.492587 | 525400d7-b628-d15b-cf41-00000000001a | TASK | Delete OVN MAC Address ports
2021-07-31 14:48:10.435116 | 525400d7-b628-d15b-cf41-00000000001a | CHANGED | Delete OVN MAC Address ports | localhost
2021-07-31 14:48:10.436616 | 525400d7-b628-d15b-cf41-00000000001a | TIMING | Delete OVN MAC Address ports | localhost | 0:00:07.410627 | 2.94s
2021-07-31 14:48:10.441848 | 525400d7-b628-d15b-cf41-00000000001b | TASK | Delete heat stack
2021-07-31 14:48:11.949180 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 16 retries left
2021-07-31 14:48:17.550764 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 15 retries left
2021-07-31 14:48:22.940710 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 14 retries left
2021-07-31 14:48:28.325408 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 13 retries left
2021-07-31 14:48:33.731354 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 12 retries left
2021-07-31 14:48:39.089831 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 11 retries left
2021-07-31 14:48:44.472890 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 10 retries left
2021-07-31 14:48:49.917528 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 9 retries left
2021-07-31 14:48:55.314939 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 8 retries left
2021-07-31 14:49:00.714950 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 7 retries left
2021-07-31 14:49:06.081579 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 6 retries left
2021-07-31 14:49:11.474882 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 5 retries left
2021-07-31 14:49:16.856428 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 4 retries left
2021-07-31 14:49:22.390894 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 3 retries left
2021-07-31 14:49:27.778057 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 2 retries left
2021-07-31 14:49:33.149988 | 525400d7-b628-d15b-cf41-00000000001b | RETRY | Delete heat stack | localhost | 1 retries left
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: keystoneauth1.exceptions.catalog.EndpointNotFound: public endpoint for orchestration service in regionOne region not found
2021-07-31 14:49:38.511885 | 525400d7-b628-d15b-cf41-00000000001b | FATAL | Delete heat stack | localhost | error={"attempts": 16, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\",
line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy
.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.stack_payload_a2ipxtqs/ansible_openstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/modules/stack.py\", line 248, in <module>\n File \"/tmp/ansible_openstack.cloud.stack_payload_a2ipxtqs/ansible_o
penstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/modules/stack.py\", line 244, in main\n File \"/tmp/ansible_openstack.cloud.stack_payload_a2ipxtqs/ansible_openstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 406, in __call__\n Fi
le \"/tmp/ansible_openstack.cloud.stack_payload_a2ipxtqs/ansible_openstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/modules/stack.py\", line 218, in run\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_orchestration.py\", line 255, in get_stack\n self, _search_one_stack, name_o
r_id, filters)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_utils.py\", line 198, in _get_entity\n entities = search(name_or_id, filters, **kwargs)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_orchestration.py\", line 243, in _search_one_stack\n stack = self.orchestration.find_stack
(\n File \"/usr/lib/python3.6/site-packages/openstack/service_description.py\", line 87, in __get__\n proxy = self._make_proxy(instance)\n File \"/usr/lib/python3.6/site-packages/openstack/service_description.py\", line 262, in _make_proxy\n found_version = temp_adapter.get_api_major_version()\n File \"/usr/l
ib/python3.6/site-packages/keystoneauth1/adapter.py\", line 354, in get_api_major_version\n return self.session.get_api_major_version(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1276, in get_api_major_version\n return auth.get_api_major_version(self, *
*kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 500, in get_api_major_version\n data = get_endpoint_data(discover_versions=discover_versions)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 279, in get_endpoint_data\n service_name=servi
ce_name)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/access/service_catalog.py\", line 462, in endpoint_data_for\n raise exceptions.EndpointNotFound(msg)\nkeystoneauth1.exceptions.catalog.EndpointNotFound: public endpoint for orchestration service in regionOne region not found\n", "module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
2021-07-31 14:49:38.512830 | 525400d7-b628-d15b-cf41-00000000001b | TIMING | Delete heat stack | localhost | 0:01:35.486835 | 88.07s

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost : ok=7 changed=3 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0
2021-07-31 14:49:38.516503 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-07-31 14:49:38.516991 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 11 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Tags:

Michele Baldessari (michele) on 2021-07-31

Changed in tripleo:
importance:	Undecided → Medium

Revision history for this message

Brendan Shephard (bshephar) wrote on 2021-08-01:

This is currently being addressed by:
https://review.opendev.org/c/openstack/tripleo-ansible/+/800304/1/tripleo_ansible/playbooks/cli-overcloud-delete.yaml

Brendan Shephard (bshephar) on 2021-08-02

Changed in tripleo:
status:	New → In Progress

Revision history for this message

Michele Baldessari (michele) wrote on 2021-08-16:

Download full text (4.7 KiB)

Note that even though review.opendev.org/800304 merged I still get:

Note that even though review.opendev.org/800304 merged I still get:

2021-08-16 14:13:03.823567 | 5254001b-0a8c-78a9-679c-00000000001e |       TASK | Delete heat stack
2021-08-16 14:13:05.393458 | 5254001b-0a8c-78a9-679c-00000000001e |      RETRY | Delete heat stack | localhost | 3 retries left
2021-08-16 14:13:11.898565 | 5254001b-0a8c-78a9-679c-00000000001e |      RETRY | Delete heat stack | localhost | 2 retries left
2021-08-16 14:13:18.314078 | 5254001b-0a8c-78a9-679c-00000000001e |      RETRY | Delete heat stack | localhost | 1 retries left
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: keystoneauth1.exceptions.catalog.EndpointNotFound: public endpoint for orchestration service in regionOne region not found
2021-08-16 14:13:24.769580 | 5254001b-0a8c-78a9-679c-00000000001e |      FATAL | Delete heat stack | localhost | error={"attempts": 3, "changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_openstack.cloud.stack_payload_2dvf1js4/ansible_openstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/modules/stack.py\", line 248, in <module>\n  File \"/tmp/ansible_openstack.cloud.stack_payload_2dvf1js4/ansible_openstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/modules/stack.py\", line 244, in main\n  File \"/tmp/ansible_openstack.cloud.stack_payload_2dvf1js4/ansible_openstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 406, in __call__\n  File \"/tmp/ansible_openstack.cloud.stack_payload_2dvf1js4/ansible_openstack.cloud.stack_payload.zip/ansible_collections/openstack/cloud/plugins/modules/stack.py\", line 218, in run\n  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_orchestration.py\", line 255, in get_stack\n    self, _search_one_stack, name_or_id, filters)\n  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_utils.py\", line 198, in _get_entity\n    entities = search(name_or_id, filters, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_orchestration.py\", line 243, in _search_one_stack\n    stack = self.orchestration.find_stack(\n  File \"/usr/lib/python3.6/site-packages/openstack/service_description.py\", line 87, in __get__\n    proxy = self._make_proxy(instance)\n  File \"/usr/lib/python3.6/site-packages/openstack/service_description.py\", line 262, in _make_proxy\n    found_version = temp_adapter.get_api_major_version()\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 354, in get_api_major_version\n    return self.session.get_api_major_version(auth or self.auth, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1276, in get_api_major_version\n    return auth.get_api_major_version(self, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 500, in get_api_major_version\n    data = get_endpoint_data(discover_versions=discover_versions)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 279, in get_endpoint_data\n    service_name=service_name)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/access/service_catalog.py\", line 462, in endpoint_data_for\n    raise exceptions.EndpointNotFound(msg)\nkeystoneauth1.exceptions.catalog.EndpointNotFound: public endpoint for orchestration service in regionOne region not found\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
2021-08-16 14:13:24.770286 | 5254001b-0a8c-78a9-679c-00000000001e |     TIMING | Delete heat stack | localhost | 0:00:29.326839 | 20.95s
2021-08-16 14:13:24.775090 | 5254001b-0a8c-78a9-679c-00000000001f |       TASK | Notice
2021-08-16 14:13:24.812043 | 5254001b-0a8c-78a9-679c-00000000001f |         OK | Notice | localhost | result={
    "changed": false,
    "msg": "The Heat stack for \"overcloud\" was not deleted, depending on the deployment type this may be normal."
}
2021-08-16 14:13:24.813250 | 5254001b-0a8c-78a9-679c-00000000001f |     TIMING | Notice | localhost | 0:00:29.369808 | 0.04s

I get that afterwards we print "this might be okay", but this is a bit hostile for the user. Can't we hide that scary error message when we detect ephemeral heat at play?

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-11: Related fix proposed to tripleo-ansible (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/808537

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-11: Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/808539

Sandeep Yadav (sandeepyadav93) on 2021-09-11

Changed in tripleo:
assignee:	nobody → Sandeep Yadav (sandeepyadav93)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-13: Change abandoned on tripleo-ansible (master)

Change abandoned by "Sandeep Yadav <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/808537
Reason: To stop from merging as we need more discussion on comment's suggested by reviewers on different approach.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-13: Related fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/808537
Committed: https://opendev.org/openstack/tripleo-ansible/commit/5993686014599c0462cf5162e998b74cf630dbcb
Submitter: "Zuul (22348)"
Branch: master

commit 5993686014599c0462cf5162e998b74cf630dbcb
Author: Sandeep Yadav <email address hidden>
Date: Sat Sep 11 10:30:40 2021 +0530

Don't run stack delete in overcloud delete

    Currently in "overcloud delete" we try to delete heat stack,
    In patch[1] we added a rescue block for "Delete heat stack" task
    to not fail even if we are not able to delete a stack to handle
    the new ephermal heat stack case.

    But user still get an intermidiate keystone exception about heat
    endpoints not available[2], afterwards we print "this might be okay"
    based on deployment, but this is a bit hostile for the user .

Also for ephermal heat case, we can speed up the "overcloud delete"
execution by not trying heat stack delete at all.

    With tripleoclient patch[3] adding a new argument `--heat-type`
    in "overcloud delete" command that will default to `pod` (ephermal
    heat) and based on input we will set a var `heat_stack_delete`
    that will decide whether to run heat stack delete at all.

Reverting the original "Delete heat stack" task prior to [1]
with a conditional when based on `heat_stack_delete` var.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/807264
    [2] https://bugs.launchpad.net/tripleo/+bug/1938618/comments/2
    [3] https://review.opendev.org/c/openstack/python-tripleoclient/+/808539

Related-bug: #1938618
Change-Id: I9c7b11df6ac9a6a58bf3f4175d94095e0ef23fcf

Revision history for this message

Noam Angel (noama) wrote on 2021-09-14:

Can you backport to wallaby?

tags:

added: wallaby-backport-potential

Revision history for this message

Sandeep Yadav (sandeepyadav93) wrote on 2021-09-14:

Hello Noam,

Thanks, Yes we will backport to wallaby once patches merge in master.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-14: Related fix proposed to tripleo-ansible (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/808971

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-14: Fix merged to python-tripleoclient (master)

#10

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/808539
Committed: https://opendev.org/openstack/python-tripleoclient/commit/1d2c6981aa33c13009e60e3b94bc0430f1c39b6c
Submitter: "Zuul (22348)"
Branch: master

commit 1d2c6981aa33c13009e60e3b94bc0430f1c39b6c
Author: Sandeep Yadav <email address hidden>
Date: Sat Sep 11 11:07:38 2021 +0530

Add `--heat-type` in overcloud delete

    With this patch adding a new argument `--heat-type`
    in "overcloud delete" command that will default to `pod` (ephermal
    heat) and based on input we will set a var `heat_stack_delete`
    which will be passed to cli-overcloud-delete.yaml to decide whether
    to run heat stack delete at all.

Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/808537

Closes-bug: #1938618
Change-Id: I74cdac5b96a10ec8baefdf3080fd34e03d8f1933

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix proposed to python-tripleoclient (stable/wallaby)

#11

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/808984

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Related fix merged to tripleo-ansible (stable/wallaby)

#12

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/808971
Committed: https://opendev.org/openstack/tripleo-ansible/commit/52638174530d86927e6393fc9f72e36716695d05
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 52638174530d86927e6393fc9f72e36716695d05
Author: Sandeep Yadav <email address hidden>
Date: Sat Sep 11 10:30:40 2021 +0530

Don't run stack delete in overcloud delete

Also for ephermal heat case, we can speed up the "overcloud delete"
execution by not trying heat stack delete at all.

Reverting the original "Delete heat stack" task prior to [1]
with a conditional when based on `heat_stack_delete` var.

    Related-bug: #1938618
    Change-Id: I9c7b11df6ac9a6a58bf3f4175d94095e0ef23fcf
    (cherry picked from commit 5993686014599c0462cf5162e998b74cf630dbcb)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix merged to python-tripleoclient (stable/wallaby)

#13

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/808984
Committed: https://opendev.org/openstack/python-tripleoclient/commit/592c793fd712f9bb434ffb57d7f5c82e1c0e6af2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 592c793fd712f9bb434ffb57d7f5c82e1c0e6af2
Author: Sandeep Yadav <email address hidden>
Date: Sat Sep 11 11:07:38 2021 +0530

Add `--heat-type` in overcloud delete

Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/808537

    Closes-bug: #1938618
    Change-Id: I74cdac5b96a10ec8baefdf3080fd34e03d8f1933
    (cherry picked from commit 1d2c6981aa33c13009e60e3b94bc0430f1c39b6c)