I reproduced this issue running this test: FAIL 20210518 12:30:47 testcases/sanity/sanity_openstack/test_host_management.py::test_reapply_openstack
Debug log: controller-0:~$ cat /var/log/armada/stx-openstack-delete_2021-05-18-12-30-07.log 2021-05-18 12:30:09.235 177 DEBUG armada.handlers.tiller [-] Using Tiller host IP: 127.0.0.1 _get_tiller_ip /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165 2021-05-18 12:30:09.235 177 DEBUG armada.handlers.tiller [-] Using Tiller host port: 24134 _get_tiller_port /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:174 2021-05-18 12:30:09.235 177 DEBUG armada.handlers.tiller [-] Tiller getting gRPC insecure channel at 127.0.0.1:24134 with options: [grpc.max_send_message_length=429496729, grpc.max_receive_message_length=429496729] get_channel /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:127 2021-05-18 12:30:09.238 177 DEBUG armada.handlers.tiller [-] Armada is using Tiller at: 127.0.0.1:24134, namespace=kube-system, timeout=300 __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:107 2021-05-18 12:30:09.238 177 INFO armada.handlers.lock [-] Acquiring lock 2021-05-18 12:30:09.250 177 DEBUG armada.handlers.tiller [-] Getting known releases from Tiller... list_charts /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:359 2021-05-18 12:30:09.250 177 DEBUG armada.handlers.tiller [-] Tiller ListReleases() with timeout=300, request=limit: 32 status_codes: UNKNOWN status_codes: DEPLOYED status_codes: DELETED status_codes: DELETING status_codes: FAILED status_codes: PENDING_INSTALL status_codes: PENDING_UPGRADE status_codes: PENDING_ROLLBACK get_results /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:215 2021-05-18 12:30:10.251 177 INFO armada.handlers.lock [-] Releasing lock 2021-05-18 12:30:10.257 177 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.UNKNOWN details = "write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe" debug_error_string = "{"created":"@1621341009.251408267","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe","grpc_status":2}" > 2021-05-18 12:30:10.257 177 ERROR armada.cli Traceback (most recent call last): 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke 2021-05-18 12:30:10.257 177 ERROR armada.cli self.invoke() 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/delete.py", line 98, in invoke 2021-05-18 12:30:10.257 177 ERROR armada.cli self.handle(tiller) 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper 2021-05-18 12:30:10.257 177 ERROR armada.cli return future.result() 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result 2021-05-18 12:30:10.257 177 ERROR armada.cli return self.__get_result() 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-05-18 12:30:10.257 177 ERROR armada.cli raise self._exception 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2021-05-18 12:30:10.257 177 ERROR armada.cli result = self.fn(*self.args, **self.kwargs) 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/delete.py", line 102, in handle 2021-05-18 12:30:10.257 177 ERROR armada.cli known_release_names = [release[0] for release in tiller.list_charts()] 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 361, in list_charts 2021-05-18 12:30:10.257 177 ERROR armada.cli for latest_release in self.list_releases(): 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 252, in list_releases 2021-05-18 12:30:10.257 177 ERROR armada.cli releases = get_results() 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 220, in get_results 2021-05-18 12:30:10.257 177 ERROR armada.cli for message in response: 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 364, in __next__ 2021-05-18 12:30:10.257 177 ERROR armada.cli return self._next() 2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next 2021-05-18 12:30:10.257 177 ERROR armada.cli raise self 2021-05-18 12:30:10.257 177 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: 2021-05-18 12:30:10.257 177 ERROR armada.cli status = StatusCode.UNKNOWN 2021-05-18 12:30:10.257 177 ERROR armada.cli details = "write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe" 2021-05-18 12:30:10.257 177 ERROR armada.cli debug_error_string = "{"created":"@1621341009.251408267","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe","grpc_status":2}" 2021-05-18 12:30:10.257 177 ERROR armada.cli > 2021-05-18 12:30:10.257 177 ERROR armada.cli command terminated with exit code 1 controller-0:~$
I reproduced this issue running this test: sanity/ sanity_ openstack/ test_host_ management. py::test_ reapply_ openstack
FAIL 20210518 12:30:47 testcases/
Debug log: armada/ stx-openstack- delete_ 2021-05- 18-12-30- 07.log handlers. tiller [-] Using Tiller host IP: 127.0.0.1 _get_tiller_ip /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:165 handlers. tiller [-] Using Tiller host port: 24134 _get_tiller_port /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:174 handlers. tiller [-] Tiller getting gRPC insecure channel at 127.0.0.1:24134 with options: [grpc.max_ send_message_ length= 429496729, grpc.max_ receive_ message_ length= 429496729] get_channel /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:127 handlers. tiller [-] Armada is using Tiller at: 127.0.0.1:24134, namespace= kube-system, timeout=300 __init__ /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:107 handlers. lock [-] Acquiring lock handlers. tiller [-] Getting known releases from Tiller... list_charts /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:359 handlers. tiller [-] Tiller ListReleases() with timeout=300, request=limit: 32 lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:215 handlers. lock [-] Releasing lock _Rendezvous: <_Rendezvous of RPC that terminated with: 166.171: 34140-> 10.10.56. 10:5432: write: broken pipe"
debug_ error_string = "{"created" :"@1621341009. 251408267" ,"description" :"Error received from peer"," file":" src/core/ lib/surface/ call.cc" ,"file_ line":1017, "grpc_message" :"write tcp 172.16. 166.171: 34140-> 10.10.56. 10:5432: write: broken pipe"," grpc_status" :2}" lib/python3. 6/dist- packages/ armada/ cli/__init_ _.py", line 38, in safe_invoke lib/python3. 6/dist- packages/ armada/ cli/delete. py", line 98, in invoke lib/python3. 6/dist- packages/ armada/ handlers/ lock.py" , line 81, in func_wrapper python3. 6/concurrent/ futures/ _base.py" , line 425, in result python3. 6/concurrent/ futures/ _base.py" , line 384, in __get_result python3. 6/concurrent/ futures/ thread. py", line 56, in run lib/python3. 6/dist- packages/ armada/ cli/delete. py", line 102, in handle list_charts( )] lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py", line 361, in list_charts releases( ): lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py", line 252, in list_releases lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py", line 220, in get_results lib/python3. 6/dist- packages/ grpc/_channel. py", line 364, in __next__ lib/python3. 6/dist- packages/ grpc/_channel. py", line 358, in _next _Rendezvous: <_Rendezvous of RPC that terminated with: 166.171: 34140-> 10.10.56. 10:5432: write: broken pipe" :"@1621341009. 251408267" ,"description" :"Error received from peer"," file":" src/core/ lib/surface/ call.cc" ,"file_ line":1017, "grpc_message" :"write tcp 172.16. 166.171: 34140-> 10.10.56. 10:5432: write: broken pipe"," grpc_status" :2}"
controller-0:~$ cat /var/log/
2021-05-18 12:30:09.235 177 DEBUG armada.
2021-05-18 12:30:09.235 177 DEBUG armada.
2021-05-18 12:30:09.235 177 DEBUG armada.
2021-05-18 12:30:09.238 177 DEBUG armada.
2021-05-18 12:30:09.238 177 INFO armada.
2021-05-18 12:30:09.250 177 DEBUG armada.
2021-05-18 12:30:09.250 177 DEBUG armada.
status_codes: UNKNOWN
status_codes: DEPLOYED
status_codes: DELETED
status_codes: DELETING
status_codes: FAILED
status_codes: PENDING_INSTALL
status_codes: PENDING_UPGRADE
status_codes: PENDING_ROLLBACK
get_results /usr/local/
2021-05-18 12:30:10.251 177 INFO armada.
2021-05-18 12:30:10.257 177 ERROR armada.cli [-] Caught unexpected exception: grpc._channel.
status = StatusCode.UNKNOWN
details = "write tcp 172.16.
>
2021-05-18 12:30:10.257 177 ERROR armada.cli Traceback (most recent call last):
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli self.invoke()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli self.handle(tiller)
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli return future.result()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/
2021-05-18 12:30:10.257 177 ERROR armada.cli return self.__get_result()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/
2021-05-18 12:30:10.257 177 ERROR armada.cli raise self._exception
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/
2021-05-18 12:30:10.257 177 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli known_release_names = [release[0] for release in tiller.
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli for latest_release in self.list_
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli releases = get_results()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli for message in response:
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli return self._next()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/
2021-05-18 12:30:10.257 177 ERROR armada.cli raise self
2021-05-18 12:30:10.257 177 ERROR armada.cli grpc._channel.
2021-05-18 12:30:10.257 177 ERROR armada.cli status = StatusCode.UNKNOWN
2021-05-18 12:30:10.257 177 ERROR armada.cli details = "write tcp 172.16.
2021-05-18 12:30:10.257 177 ERROR armada.cli debug_error_string = "{"created"
2021-05-18 12:30:10.257 177 ERROR armada.cli >
2021-05-18 12:30:10.257 177 ERROR armada.cli
command terminated with exit code 1
controller-0:~$