Platform: cert-manager apply failed

[Yvonne Ding]

Brief Description
-----------------
cert-manager apply failed. Unexpected exception of "write tcp 172.16.192.79:32904->192.168.204.2:5432: write: connection timed out" is thrown.

Severity
--------
Major

Steps to Reproduce
------------------
platform storage regression

Expected Behavior
------------------
cert-manager apply success

Actual Behavior
----------------
cert-manager apply failed

Reproducibility
---------------
reproducible

System Configuration
--------------------
Storage 2+4+2

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20210318T063229Z"

Timestamp/Logs
--------------
2021-03-20 06:31:33.255 321 INFO armada.handlers.lock [-] Releasing lock^[[00m
2021-03-20 06:31:33.262 321 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "write tcp 172.16.192.79:32904->192.168.204.2:5432: write: connection timed out"
        debug_error_string = "{"created":"@1616221892.284463009","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"write tcp 172.16.192.79:32904->192.168.204.2:5432: write: connection timed out","grpc_status":2}"
>
2021-03-20 06:31:33.262 321 ERROR armada.cli Traceback (most recent call last):
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2021-03-20 06:31:33.262 321 ERROR armada.cli self.invoke()
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2021-03-20 06:31:33.262 321 ERROR armada.cli resp = self.handle(documents, tiller)
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2021-03-20 06:31:33.262 321 ERROR armada.cli return future.result()
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-03-20 06:31:33.262 321 ERROR armada.cli return self.__get_result()
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-03-20 06:31:33.262 321 ERROR armada.cli raise self._exception
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-03-20 06:31:33.262 321 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2021-03-20 06:31:33.262 321 ERROR armada.cli return armada.sync()
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 189, in sync
2021-03-20 06:31:33.262 321 ERROR armada.cli known_releases = self.tiller.list_releases()
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 252, in list_releases
2021-03-20 06:31:33.262 321 ERROR armada.cli releases = get_results()
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 220, in get_results
2021-03-20 06:31:33.262 321 ERROR armada.cli for message in response:
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 364, in __next__
2021-03-20 06:31:33.262 321 ERROR armada.cli return self._next()
2021-03-20 06:31:33.262 321 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next
2021-03-20 06:31:33.262 321 ERROR armada.cli raise self
2021-03-20 06:31:33.262 321 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2021-03-20 06:31:33.262 321 ERROR armada.cli status = StatusCode.UNKNOWN
2021-03-20 06:31:33.262 321 ERROR armada.cli details = "write tcp 172.16.192.79:32904->192.168.204.2:5432: write: connection timed out"
2021-03-20 06:31:33.262 321 ERROR armada.cli debug_error_string = "{"created":"@1616221892.284463009","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"write tcp 172.16.192.79:32904->192.168.204.2:5432: write: connection timed out","grpc_status":2}"
2021-03-20 06:31:33.262 321 ERROR armada.cli >
2021-03-20 06:31:33.262 321 ERROR armada.cli ^[[00m
command terminated with exit code 1

[Ghada Khalil (gkhalil)]

screening: this looks like an armada/containerized application framework issue as there's been no changes to cert-mgr since it's introduction in stx.4.0

[Ghada Khalil (gkhalil)]

failure signature appears similar to https://bugs.launchpad.net/starlingx/+bug/1923587

[Andrei Grosu (agrosu1)]

Seems a duplicate of : https://bugs.launchpad.net/starlingx/+bug/1923587

[Yvonne Ding (yding)]

@Frank, could you please duplicate https://bugs.launchpad.net/starlingx/+bug/1923587 to this LP?
This LP was created earlier than https://bugs.launchpad.net/starlingx/+bug/1923587.