Distributed Cloud Ipv6: application reapply failed by "Name resolution failure"

[Peng Peng]

Brief Description
-----------------
platform-integ-apps application initially applied success. Re-apply it failed by "Name resolution failure"

Severity
--------
Major

Steps to Reproduce
------------------
as description

TC-name:

Expected Behavior
------------------

Actual Behavior
----------------

Reproducibility
---------------
Seen once

System Configuration
--------------------
DC
IPv6

Lab-name: DC

Branch/Pull Time/Commit
-----------------------
2019-11-21_20-00-00

Last Pass
---------
unknown

Timestamp/Logs
--------------
2019-11-27 19:52:26.159 341 INFO armada.handlers.lock [-] Releasing lock
2019-11-27 19:52:26.164 341 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Name resolution failure"
        debug_error_string = "{"created":"@1574884345.171711560","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2721,"referenced_errors":[{"created":"@1574884345.171709189","description":"Name resolution failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3026,"grpc_status":14}]}"
>
2019-11-27 19:52:26.164 341 ERROR armada.cli Traceback (most recent call last):
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2019-11-27 19:52:26.164 341 ERROR armada.cli self.invoke()
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2019-11-27 19:52:26.164 341 ERROR armada.cli resp = self.handle(documents, tiller)
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2019-11-27 19:52:26.164 341 ERROR armada.cli return future.result()
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2019-11-27 19:52:26.164 341 ERROR armada.cli return self.__get_result()
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2019-11-27 19:52:26.164 341 ERROR armada.cli raise self._exception
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2019-11-27 19:52:26.164 341 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2019-11-27 19:52:26.164 341 ERROR armada.cli return armada.sync()
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 189, in sync
2019-11-27 19:52:26.164 341 ERROR armada.cli known_releases = self.tiller.list_releases()
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 252, in list_releases
2019-11-27 19:52:26.164 341 ERROR armada.cli releases = get_results()
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 220, in get_results
2019-11-27 19:52:26.164 341 ERROR armada.cli for message in response:
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 364, in __next__
2019-11-27 19:52:26.164 341 ERROR armada.cli return self._next()
2019-11-27 19:52:26.164 341 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next
2019-11-27 19:52:26.164 341 ERROR armada.cli raise self
2019-11-27 19:52:26.164 341 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2019-11-27 19:52:26.164 341 ERROR armada.cli status = StatusCode.UNAVAILABLE
2019-11-27 19:52:26.164 341 ERROR armada.cli details = "Name resolution failure"
2019-11-27 19:52:26.164 341 ERROR armada.cli debug_error_string = "{"created":"@1574884345.171711560","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2721,"referenced_errors":[{"created":"@1574884345.171709189","description":"Name resolution failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3026,"grpc_status":14}]}"
2019-11-27 19:52:26.164 341 ERROR armada.cli >
2019-11-27 19:52:26.164 341 ERROR armada.cli

Test Activity
-------------
Regression Testing

[Peng Peng (ppeng)]

The issue seems different from 1849688, since reapply workaround is not working.

[Ghada Khalil (gkhalil)]

Yang Liu suspects that this was due lab issues. @Peng, please re-test and let us know if the issue is reproducible.

[Peng Peng (ppeng)]

Issue was not reproduced on "2019-12-01_20-00-00"

[Ghada Khalil (gkhalil)]

Closing; Issue is not reproducible