stx-openstack: fails to remove because of BackoffLimitExceeded error

[Rafael Falcão]

Brief Description
-----------------
stx-openstack reaches remove-failed status when its removal is triggered.

Severity
--------
Major

Steps to Reproduce
------------------
1 - system application-upload stx-openstack.tgz
2 - system application-apply stx-openstack
3 - system application-remove stx-openstack

Expected Behavior
------------------
stx-openstack reaches the uploaded status

Actual Behavior
----------------
stx-openstack reaches the remove-failed status

Reproducibility
---------------
Reproducible

System Configuration
--------------------
N/A

Timestamp/Logs
--------------
/var/log/armada/<deletion-log>
2022-07-19 10:05:35.100 262 ERROR armada.cli armada.exceptions.tiller_exceptions.ReleaseException: Failed to Delete release: osh-openstack-neutron - Tiller Message: b'Install complete'

kubectl logs -f -c tiller -n armada <armada-pod>
[tiller] 2022/07/19 10:51:42 warning: Release osh-openstack-neutron pre-delete neutron/job-resources-cleanup.yaml could not complete: Job failed: BackoffLimitExceeded
[tiller] 2022/07/19 10:51:42 deleting pre-delete hook neutron-resources-cleanup for release osh-openstack-neutron due to "hook-failed" policy

Test Activity
-------------
Developer testing

Workaround
----------
Manually remove each stx-openstack resource, update stx-openstack status to 'uploaded' direct to the database, trigger the deletion of the stx-openstack

[Ghada Khalil (gkhalil)]

screening: issue found during the investigation of https://bugs.launchpad.net/starlingx/+bug/1980397 Community agreed that this should be gating for stx.7.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/850765

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/850765
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/81145bca935548f82dc36f58a2b916bf4d07f820
Submitter: "Zuul (22348)"
Branch: master

commit 81145bca935548f82dc36f58a2b916bf4d07f820
Author: Rafael Falcão <email address hidden>
Date: Fri Jul 22 12:30:24 2022 -0300

    Adding trunk as neutron service plugin

    In order to the Network Resources Cleanup (added in
    b5b4cc562a4c7d705d108a8280dc1b246159d90b) successfully performs
    its operations, it is necessary to add trunk as one of the neutron's
    service plugins. This review adds a new service plugin for neutron
    to allow stx-openstack to be successfully removed.

    Test Plan:
    PASS - Build stx-openstack armada tarball (CentOS)
    PASS - Upload stx-openstack on Stx7.0/master build
    PASS - Apply stx-openstck on a Stx7.0/master build
    PASS - Remove stx-openstack on a Stx7.0/master build
    PASS - Delete stx-openstack on a Stx7.0/master build

    Closes-bug: 1982220

    Signed-off-by: Rafael Falcao <email address hidden>
    Change-Id: I2590fdcf7b9551a94757ca04e2dbc11640967508

[Ghada Khalil (gkhalil)]

@Rafael, can you please check if this commit made it in the r/stx.7.0 release branch? The cutoff time was July 26 1pm, so it's pretty close. If it's not there, please cherry-pick it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.7.0
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/851517

[Rafael Falcão (rafaelvfalc)]

@Ghada I just created the review with the cherrypick to the r/stx.7.0 branch. https://review.opendev.org/c/starlingx/openstack-armada-app/+/851518

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rafael Vieira Falcão <email address hidden>" on branch: r/stx.7.0
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/851517
Reason: Forgot to update the change-id

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.7.0
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/851518

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/851518
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/66511c1833de6444b0468753576c1c98e81b62a2
Submitter: "Zuul (22348)"
Branch: r/stx.7.0

commit 66511c1833de6444b0468753576c1c98e81b62a2
Author: Rafael Falcão <email address hidden>
Date: Fri Jul 22 12:30:24 2022 -0300

    Adding trunk as neutron service plugin

    In order to the Network Resources Cleanup (added in
    b5b4cc562a4c7d705d108a8280dc1b246159d90b) successfully performs
    its operations, it is necessary to add trunk as one of the neutron's
    service plugins. This review adds a new service plugin for neutron
    to allow stx-openstack to be successfully removed.

    Test Plan:
    PASS - Build stx-openstack armada tarball (CentOS)
    PASS - Upload stx-openstack on Stx7.0/master build
    PASS - Apply stx-openstck on a Stx7.0/master build
    PASS - Remove stx-openstack on a Stx7.0/master build
    PASS - Delete stx-openstack on a Stx7.0/master build

    Closes-bug: 1982220

    Signed-off-by: Rafael Falcao <email address hidden>
    (cherry picked from commit 81145bca935548f82dc36f58a2b916bf4d07f820)
    Change-Id: I0dfdc0a0d95b5d7322647ff481f11ed49d5d9167