system application-remove oidc-auth-apps failed due to "Name resolution failure"

[Difu Hu]

Brief Description
-----------------
system application-remove oidc-auth-apps failed due to "Name resolution failure" in armada

Severity
--------
Major

Steps to Reproduce
------------------
system application-apply oidc-auth-apps
system application-remove oidc-auth-apps

Expected Behavior
------------------
oidc-auth-apps is in uploaded/completed status.

Actual Behavior
----------------
oidc-auth-apps is in remove-failed status

Reproducibility
---------------
Intermittent - happened 1 times on one specific system

System Configuration
--------------------
Lab-name: DC-4 controller

Branch/Pull Time/Commit
-----------------------
2020-06-27_18-35-20

Last Pass
---------
not sure

Timestamp/Logs
+--------------------------+----------+-----------------------------------+----------------------------------+---------------+-----------------------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+----------+-----------------------------------+----------------------------------+---------------+-----------------------+
| oidc-auth-apps | 20.06-26 | oidc-auth-manifest | manifest.yaml | remove-failed | operation aborted, |
| | | | | | check logs for detail |

2020-07-05 03:39:19.823 269 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next
2020-07-05 03:39:19.823 269 ERROR armada.cli raise self
2020-07-05 03:39:19.823 269 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2020-07-05 03:39:19.823 269 ERROR armada.cli status = StatusCode.UNAVAILABLE
2020-07-05 03:39:19.823 269 ERROR armada.cli details = "Name resolution failure"
2020-07-05 03:39:19.823 269 ERROR armada.cli debug_error_string = "{"created":"@1593920358.850017178","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2721,"referenced_errors":[{"created":"@1593920358.850010816","description":"Name resolution failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3026,"grpc_status":14}]}"

Test Activity
-------------
Regression Testing

[Difu Hu (difuhu)]

Manually remove oidc-auth-apps succeeded.

[Difu Hu (difuhu)]

log
https://files.starlingx.kube.cengn.ca/launchpad/1886533

[Ghada Khalil (gkhalil)]

Marking as low priority - this is not a likely user scenario. If a user needs the oidc app, they won't remove it. Also retrying the remove seems to work as per note above.

[Ghada Khalil (gkhalil)]

Assigning to Dan Voiculeasa as he has other LPs with the same failure signature: Failed to create subchannel

[[e-mail] (r-sdmin-0)]

I have the same error in applying stx-openstack.

I tried to modify controller-0's /etc/resolv.conf and restart armada via docker restart.

Anyone have idea how to troubleshoot and resolve this?