stx-openstack apply failed: Failed to delete job heat-engine-cleaner-1620392400

[Alexandru Dimofte]

Brief Description
-----------------
stx-openstack apply-failed because of failure in deleting the job heat-engine-cleaner-1620392400:
2021-05-07 13:15:31.047 135 INFO armada.handlers.tiller [-] [chart=openstack-heat]: Deleting job heat-engine-cleaner-1620392400 in namespace: openstack
2021-05-07 13:15:31.047 135 DEBUG armada.handlers.k8s [-] [chart=openstack-heat]: Watching to delete job heat-engine-cleaner-1620392400, Wait timeout=1799 _delete_item_action /usr/local/lib/python3.6/dist-packages/armada/handlers/k8s.py:155
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s [-] [chart=openstack-heat]: Exception when deleting job: name=heat-engine-cleaner-1620392400, namespace=openstack: kubernetes.client.rest.ApiException: (404)
Reason: Not Found

Severity
--------
<Minor: System/Feature is usable with minor issue>

Steps to Reproduce
------------------
This happened during sanity execution. I didn't identified yet the problematic sequence.

Expected Behavior
------------------
stx-openstack should apply fine.

Actual Behavior
----------------
stx-openstack apply failed.

2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s Traceback (most recent call last):
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/armada/handlers/k8s.py", line 167, in _delete_item_action
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s propagation_policy=propagation_policy)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/batch_v1_api.py", line 306, in delete_namespaced_job
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s (data) = self.delete_namespaced_job_with_http_info(name, namespace, **kwargs)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/batch_v1_api.py", line 406, in delete_namespaced_job_with_http_info
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s collection_formats=collection_formats)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 334, in call_api
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s _return_http_data_only, collection_formats, _preload_content, _request_timeout)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 168, in __call_api
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s _request_timeout=_request_timeout)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 400, in request
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s body=body)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 256, in DELETE
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s body=body)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 222, in request
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s raise ApiException(http_resp=r)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s kubernetes.client.rest.ApiException: (404)
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s Reason: Not Found
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Fri, 07 May 2021 13:15:31 GMT', 'Content-Length': '250'})
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"heat-engine-cleaner-1620392400\" not found","reason":"NotFound","details":{"name":"heat-engine-cleaner-1620392400","group":"batch","kind":"jobs"},"code":404}
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s
2021-05-07 13:15:31.083 135 ERROR armada.handlers.k8s
2021-05-07 13:15:31.091 135 ERROR armada.handlers.tiller [-] [chart=openstack-heat]: Pre-action failure: could not delete job None.: kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Fri, 07 May 2021 13:15:31 GMT', 'Content-Length': '250'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"heat-engine-cleaner-1620392400\" not found","reason":"NotFound","details":{"name":"heat-engine-cleaner-1620392400","group":"batch","kind":"jobs"},"code":404}

Reproducibility
---------------
Till now I saw this issue once on Duplex baremetal.

System Configuration
Two node system

Branch/Pull Time/Commit
-----------------------
RC 5.0
Last Pass
---------
20210505T230338Z/

Timestamp/Logs
--------------
will be attached

Test Activity
-------------
Sanity

Workaround
----------
-

[Austin Sun (sunausti)]

aligned w/ Alex, this issue is not reproduced recently , put to low and keep monitor, if not be seen , will close later.

[Thales Elero Cervi (tcervi)]

Closing it now due to inactivity.