unable to delete stack without manually removing VIPs

[John Fulton]

After deploying an overcloud and passing a VipSubnetMap [1] I was unable to run `openstack overcloud delete $STACK` because the Heat stack would get into status DELETE_FAILED.

I was able to workaround it [2] by manually deleting the ports for ovn_dbs_virtual_ip and redis_virtual_ip [3]

[1]
parameter_defaults:
    VipSubnetMap:
        ctlplane: ctlplane-subnet
        InternalApi: internal_api_cloud_0_subnet
        Storage: storage_cloud_0_subnet
        StorageMgmt: storage_mgmt_cloud_0_subnet
        Tenant: tenant_cloud_0_subnet
        External: external_cloud_0_subnet
        redis: internal_api_cloud_0_subnet
        ovn_dbs: internal_api_cloud_0_subnet

[2] http://paste.openstack.org/show/805307/

[3]
openstack port delete ovn_dbs_virtual_ip
openstack port delete redis_virtual_ip

[John Fulton (jfulton-org)]

I confirm I have the following in my undercloud:

 https://github.com/openstack/tripleo-ansible/commit/812383a42918c612b829c54941caa27118a224a0
 https://github.com/openstack/tripleo-ansible/commit/de8bfa8048a6005ea5af6f61d735edca5021e3eb
 https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/playbooks/cli-overcloud-delete.yaml#L33-L36

[John Fulton (jfulton-org)]

2021-05-14 10:44:29,480 p=559929 u=stack n=ansible | PLAY [Overcloud Delete] ********************************************************
2021-05-14 10:44:29,487 p=559929 u=stack n=ansible | 2021-05-14 10:44:29.485510 | 244201a4-5bd8-63f2-c8fa-000000000015 | TASK | Check for required inputs
2021-05-14 10:44:29,513 p=559929 u=stack n=ansible | 2021-05-14 10:44:29.512832 | 244201a4-5bd8-63f2-c8fa-000000000015 | SKIPPED | Check for required inputs | localhost | item=stack_name
2021-05-14 10:44:29,518 p=559929 u=stack n=ansible | 2021-05-14 10:44:29.518385 | 244201a4-5bd8-63f2-c8fa-000000000019 | TASK | Delete service vips used
2021-05-14 10:44:30,998 p=559929 u=stack n=ansible | 2021-05-14 10:44:30.997686 | 244201a4-5bd8-63f2-c8fa-000000000019 | CHANGED | Delete service vips used | localhost
2021-05-14 10:44:31,003 p=559929 u=stack n=ansible | 2021-05-14 10:44:31.002993 | 244201a4-5bd8-63f2-c8fa-00000000001a | TASK | Delete heat stack
2021-05-14 10:59:35,947 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.947025 | 244201a4-5bd8-63f2-c8fa-00000000001a | CHANGED | Delete heat stack | localhost
2021-05-14 10:59:35,953 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.952985 | 244201a4-5bd8-63f2-c8fa-000000000017 | TASK | Workflow notice
2021-05-14 10:59:35,981 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.980961 | 244201a4-5bd8-63f2-c8fa-000000000017 | OK | Workflow notice | localhost | result={
    "changed": false,
    "msg": "Ansible workflow completed."
}
2021-05-14 10:59:35,983 p=559929 u=stack n=ansible | PLAY RECAP *********************************************************************
2021-05-14 10:59:35,984 p=559929 u=stack n=ansible | localhost : ok=4 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
2021-05-14 10:59:35,985 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.984930 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-05-14 10:59:35,985 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.985285 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 5 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-05-14 10:59:35,985 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.985695 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:15:06.863804 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-05-14 10:59:35,986 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.986008 | UUID | Info | Host | Task Name | Run Time
2021-05-14 10:59:35,986 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.986325 | 244201a4-5bd8-63f2-c8fa-00000000001a | SUMMARY | localhost | Delete heat stack | 904.94s
2021-05-14 10:59:35,986 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.986714 | 244201a4-5bd8-63f2-c8fa-000000000019 | SUMMARY | localhost | Delete service vips used | 1.48s
2021-05-14 10:59:35,987 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.987089 | 244201a4-5bd8-63f2-c8fa-000000000009 | SUMMARY | localhost | Check if undercloud is an ipa client | 0.30s
2021-05-14 10:59:35,987 p=559929 u=stack n=ansible | 2021-05-14 10:59:35.987426 | 244201a4-5bd8-63f2-c8fa-000000000015 | SUMMARY | localho...

[John Fulton (jfulton-org)]

In comment #2 you see

"TASK | Delete service vips used" completed without error in 1.48 seconds.
"TASK | Delete heat stack" took 904.94 seconds (15 minutes) because it was failing and retrying.

After I manually deleted the ports it succeeded quickly with the stack delete.

[Rabi Mishra (rabi)]

I just tested again and it seems to work for me.

Before deleting overcloud-0

(undercloud) [CentOS-8 - stack@undercloud ~]$ openstack port list | grep virtual_ip
| 1e282d87-48ec-4a1c-aed7-db2e2738a2db | internal_api_cloud_0_virtual_ip | fa:16:3e:e0:a8:ae | ip_address='172.16.13.45', subnet_id='8ec940ad-d1bb-476c-b305-757dcbc8b874' | DOWN |
| 25016e27-8bae-4955-a4af-776a94587a51 | ovn_dbs_virtual_ip | fa:16:3e:0e:8a:e3 | ip_address='172.16.13.189', subnet_id='8ec940ad-d1bb-476c-b305-757dcbc8b874' | DOWN |
| 2eed420c-f92f-4a0e-b00e-bfa8aba27ae2 | redis_virtual_ip | fa:16:3e:97:d2:7b | ip_address='172.16.13.219', subnet_id='8ec940ad-d1bb-476c-b305-757dcbc8b874' | DOWN |
| 49af0ef7-6840-4cd1-9d66-fa2552794eef | external_cloud_0_virtual_ip | fa:16:3e:9e:d7:60 | ip_address='192.168.100.21', subnet_id='bb69e475-e85c-4277-906b-8e296c9d33ea' | DOWN |
| b491f13e-f9bd-4b0a-b1a7-f680b5547c00 | control_virtual_ip | fa:16:3e:62:52:6b | ip_address='192.168.24.11', subnet_id='abc83e41-3ab3-46a6-8bf6-16029bf4af0d' | DOWN |
| c2c50e17-1745-46ed-9e95-a1efb54da806 | storage_cloud_0_virtual_ip | fa:16:3e:d3:60:8c | ip_address='172.16.11.212', subnet_id='fc7912b5-7ab7-4137-bde5-59ed54660a37' | DOWN |
| fccabd72-ca36-42b5-862e-ff121ff8b36f | storage_mgmt_cloud_0_virtual_ip | fa:16:3e:1b:c0:cc | ip_address='172.16.12.9', subnet_id='6364c82e-e5f1-4699-9655-4643b6c37514' | DOWN |

(undercloud) [CentOS-8 - stack@undercloud ~]$ openstack overcloud delete overcloud-0
Are you sure you want to delete this overcloud [y/N]? y

PLAY [delete ipa entries for overcloud nodes] **********************************
2021-05-14 12:25:58.872234 | 24420171-6e44-b5f4-35ef-000000000009 | TASK | Check if undercloud is an ipa client
2021-05-14 12:25:59.336429 | 24420171-6e44-b5f4-35ef-000000000009 | OK | Check if undercloud is an ipa client | localhost
2021-05-14 12:25:59.338512 | 24420171-6e44-b5f4-35ef-000000000009 | TIMING | Check if undercloud is an ipa client | localhost | 0:00:00.527671 | 0.47s

PLAY [Overcloud Delete] ********************************************************
2021-05-14 12:25:59.367229 | 24420171-6e44-b5f4-35ef-000000000015 | TASK | Check for required inputs
2021-05-14 12:25:59.408961 | 24420171-6e44-b5f4-35ef-000000000015 | SKIPPED | Check for required inputs | localhost | item=stack_name
2021-05-14 12:25:59.411479 | 24420171-6e44-b5f4-35ef-000000000015 | TIMING | Check for required inputs | localhost | 0:00:00.600620 | 0.04s
2021-05-14 12:25:59.419022 | 24420171-6e44-b5f4-35ef-000000000019 | TASK | Delete service vips used
2021-05-14 12:26:03.837230 | 24420171-6e44-b5f4-35ef-000000000019 | CHANGED | Delete service vips used | localhost
2021-05-14 12:26:03.838918 | 24420171-6e44-b5f4-35ef-000000000019 | TIMING | Delete service vips used | localhost | 0:00:05.028079 | 4.42s
2021-05-14 12:26:03.845620 | 24420171-6e44-b5f4-35ef-00000000001a | TASK | Delete heat stack
2021-05-14 12:29:47.419273 | 24420171-6e44-b5f4-35ef-00000000001a | CHANGED | Delete heat stack | localhost
2021-05-14 12:29:47.421707 | 24420171-6e44-b5f4-35ef-00000000001a | TI...

[Harald Jensås (harald-jensas)]

I tested as well:

(undercloud) [centos@undercloud ~]$ openstack port list | grep virtual_ip [452/452]
| 149edc8a-f82a-461c-b309-d3b476155465 | storage_mgmt_virtual_ip | fa:16:3e:d2:ed:5c | ip_address='172.19.0.160', subnet_id='cd8d64a7-5813-4d7b-8ecc-96f3a40a2333' | DOWN |
| 36a2372a-83df-4d68-8a2b-d926d6ac84d7 | control_virtual_ip | fa:16:3e:c9:10:9e | ip_address='192.168.25.25', subnet_id='18923790-a949-4821-9646-1029fc4de5c7' | DOWN |
| 78229625-3928-44bc-816b-e1d460017846 | internal_api_virtual_ip | fa:16:3e:03:12:c9 | ip_address='172.17.0.17', subnet_id='2d5d44bc-c81b-460b-9ce6-815987e8625d' | DOWN |
| 8c50e45c-5d7e-41ee-92e3-ee582d00e770 | storage_virtual_ip | fa:16:3e:62:c1:b7 | ip_address='172.18.0.210', subnet_id='a4664da3-0236-4e99-a5bc-7616757d84f3' | DOWN |
| a2f4e867-4ca8-47bd-87b2-ca67a90e27e2 | ovn_dbs_virtual_ip | fa:16:3e:c8:7c:f2 | ip_address='172.17.0.178', subnet_id='2d5d44bc-c81b-460b-9ce6-815987e8625d' | DOWN |
| a4b4c702-0870-4951-8c8a-0eefd0b9f046 | external_virtual_ip | fa:16:3e:f1:64:26 | ip_address='10.0.0.198', subnet_id='4d3a6581-ab25-49f4-8ca1-4f6196bf2b5e' | DOWN |

(undercloud) [centos@undercloud ~]$ openstack overcloud delete my_overcloud --yes

PLAY [delete ipa entries for overcloud nodes] **********************************
2021-05-14 17:18:50.922480 | fa163e92-41f8-d86d-1020-000000000009 | TASK | Check if undercloud is an ipa client
2021-05-14 17:18:51.705144 | fa163e92-41f8-d86d-1020-000000000009 | OK | Check if undercloud is an ipa client | localhost
2021-05-14 17:18:51.708583 | fa163e92-41f8-d86d-1020-000000000009 | TIMING | Check if undercloud is an ipa client | localhost | 0:00:00.939374 | 0.78s

PLAY [Overcloud Delete] ********************************************************
2021-05-14 17:18:51.754937 | fa163e92-41f8-d86d-1020-000000000015 | TASK | Check for required inputs
2021-05-14 17:18:51.828790 | fa163e92-41f8-d86d-1020-000000000015 | SKIPPED | Check for required inputs | localhost | item=stack_name
2021-05-14 17:18:51.832327 | fa163e92-41f8-d86d-1020-000000000015 | TIMING | Check for required inputs | localhost | 0:00:01.063119 | 0.07s
2021-05-14 17:18:51.844377 | fa163e92-41f8-d86d-1020-000000000019 | TASK | Delete service vips used
2021-05-14 17:18:58.056242 | fa163e92-41f8-d86d-1020-000000000019 | CHANGED | Delete service vips used | localhost
2021-05-14 17:18:58.059380 | fa163e92-41f8-d86d-1020-000000000019 | TIMING | Delete service vips used | localhost | 0:00:07.290164 | 6.21s

# ovn_dbs_virtual_ip - service VIP was deleted

(undercloud) [centos@undercloud tripleo-ansible]$ openstack port list | grep virtual_ip
| 149edc8a-f82a-461c-b309-d3b476155465 | storage_mgmt_virtual...

[John Fulton (jfulton-org)]

I am no longer able to reproduce this bug either. Since it won't reproduce it no longer seems to exist.