RDOcloud stacks cannot be deleted - taking cloud resources

Bug #1884845 reported by Ronelle Landy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Ronelle Landy

Bug Description

Since 06/17 there are stacks stuck in the openstack-nodepool that cannot be deleted by our usual means (clean scripts, manual deletes etc.). Currently we have around 23 failed stacks.

Since stuck stacks take up resources and cause failures/delays, we are logging this bug to track clean up.

Ronelle Landy (rlandy)
Changed in tripleo:
milestone: none → victoria-1
importance: Undecided → Critical
status: New → Triaged
Ronelle Landy (rlandy)
tags: added: promotion-blocker
Revision history for this message
Ronelle Landy (rlandy) wrote :

Using kforde's trick to delete some stacks - seems to be working:
heat resource-list <Stack ID> | grep -i delete | awk -F'|' '{print $3}' | xargs -n1 heat resource-list | grep -i dele | awk -F'|' '{print $3}' | xargs -n1 nova reset-state --active.

Some stacks ^^ does not work on as the server it is looking to set active does not exist.

Changed in tripleo:
assignee: nobody → Ronelle Landy (rlandy)
Revision history for this message
Ronelle Landy (rlandy) wrote :

Reset state for server ab8f2022-b8b0-45d6-b5e9-e5caa5002758 failed: No server with a name or ID of 'ab8f2022-b8b0-45d6-b5e9-e5caa5002758' exists.

What do we do about servers that are attached to stacks but do not exist to be deleted?

Revision history for this message
Ronelle Landy (rlandy) wrote :

We're left with the following:

| 525bb3d2-8edb-4187-91d6-ce29becf93a0 | baremetal_733659_28_40037 | DELETE_FAILED | 2020-06-23T19:41:01Z | 2020-06-24T01:40:07Z |
| 416adc68-ba5b-4c02-8778-e4eca0dc137e | baremetal_733165_5_33009 | DELETE_FAILED | 2020-06-23T18:37:05Z | 2020-06-24T01:41:05Z |
| 43aa8766-4900-41d1-9748-2743a2a8c21c | baremetal_727200_87_28303 | DELETE_FAILED | 2020-06-23T17:21:45Z | 2020-06-24T01:41:17Z |
| e6d820a4-0b43-486a-b959-362fb62d56a0 | baremetal_735161_2_45885 | DELETE_FAILED | 2020-06-17T03:12:40Z | 2020-06-24T01:41:28Z |
| f533b722-19d1-4e8a-be4c-d1a989f8759d | baremetal_735625_2_31198 | DELETE_FAILED | 2020-06-17T02:56:19Z | 2020-06-24T01:42:05Z |
| b66854ab-037b-4526-b834-3eb512d3194a | baremetal_736072_2_70858-extra | DELETE_FAILED | 2020-06-17T02:28:44Z | 2020-06-24T01:42:17Z |
| 906b7d50-8011-48ec-866e-d9374431b9ba | baremetal_736072_2_85164 | DELETE_FAILED | 2020-06-17T02:22:19Z | 2020-06-24T01:42:43Z |
| 2c1bcccb-463f-4741-bee3-0790feeb2099 | baremetal_60677-extra | DELETE_IN_PROGRESS | 2020-06-17T01:11:28Z | 2020-06-24T01:08:27Z |
| 60bf090b-cf0d-43c4-a54c-451568318d10 | baremetal_729358_9_8912 | DELETE_FAILED | 2020-06-17T00:56:36Z | 2020-06-24T01:43:09Z |
| 53efd6a4-2ad1-4b07-9626-66ad5c4053df | baremetal_642975_12_35988 | DELETE_FAILED | 2020-06-17T00:18:08Z | 2020-06-24T01:43:22Z |
| 666cce00-eb8e-45b8-9393-b80210ddb73d | baremetal_706365_4_7525-extra | DELETE_FAILED | 2020-02-12T11:32:20Z | 2020-06-24T01:43:34Z |
| da3718c4-97d4-4078-8d99-501b6123709f | baremetal_706550_2_78174 | DELETE_IN_PROGRESS | 2020-02-07T17:15:12Z | 2020-06-24T01:09:18Z |
| 34a7b19a-cc6c-49f7-8fa9-2bf9c0ee717a | baremetal_52528 | DELETE_COMPLETE | 2019-07-08T14:07:03Z | 2019-07-08T14:43:35Z |
| 233fdb94-645c-49f5-9f30-e1efacc13e8f | baremetal_21427-extra | DELETE_COMPLETE | 2019-02-20T17:26:54Z | 2019-02-20T23:14:07Z |

Changed in tripleo:
milestone: victoria-1 → victoria-3
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Ronelle Landy (rlandy)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.