Comment 15 for bug 1918420

Revision history for this message
Alexandru Dimofte (adimofte) wrote :

I reproduced this issue running this test:
FAIL 20210518 12:30:47 testcases/sanity/sanity_openstack/test_host_management.py::test_reapply_openstack

Debug log:
controller-0:~$ cat /var/log/armada/stx-openstack-delete_2021-05-18-12-30-07.log
2021-05-18 12:30:09.235 177 DEBUG armada.handlers.tiller [-] Using Tiller host IP: 127.0.0.1 _get_tiller_ip /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165
2021-05-18 12:30:09.235 177 DEBUG armada.handlers.tiller [-] Using Tiller host port: 24134 _get_tiller_port /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:174
2021-05-18 12:30:09.235 177 DEBUG armada.handlers.tiller [-] Tiller getting gRPC insecure channel at 127.0.0.1:24134 with options: [grpc.max_send_message_length=429496729, grpc.max_receive_message_length=429496729] get_channel /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:127
2021-05-18 12:30:09.238 177 DEBUG armada.handlers.tiller [-] Armada is using Tiller at: 127.0.0.1:24134, namespace=kube-system, timeout=300 __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:107
2021-05-18 12:30:09.238 177 INFO armada.handlers.lock [-] Acquiring lock
2021-05-18 12:30:09.250 177 DEBUG armada.handlers.tiller [-] Getting known releases from Tiller... list_charts /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:359
2021-05-18 12:30:09.250 177 DEBUG armada.handlers.tiller [-] Tiller ListReleases() with timeout=300, request=limit: 32
status_codes: UNKNOWN
status_codes: DEPLOYED
status_codes: DELETED
status_codes: DELETING
status_codes: FAILED
status_codes: PENDING_INSTALL
status_codes: PENDING_UPGRADE
status_codes: PENDING_ROLLBACK
 get_results /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:215
2021-05-18 12:30:10.251 177 INFO armada.handlers.lock [-] Releasing lock
2021-05-18 12:30:10.257 177 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe"
        debug_error_string = "{"created":"@1621341009.251408267","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe","grpc_status":2}"
>
2021-05-18 12:30:10.257 177 ERROR armada.cli Traceback (most recent call last):
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2021-05-18 12:30:10.257 177 ERROR armada.cli self.invoke()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/delete.py", line 98, in invoke
2021-05-18 12:30:10.257 177 ERROR armada.cli self.handle(tiller)
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2021-05-18 12:30:10.257 177 ERROR armada.cli return future.result()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-05-18 12:30:10.257 177 ERROR armada.cli return self.__get_result()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-05-18 12:30:10.257 177 ERROR armada.cli raise self._exception
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-05-18 12:30:10.257 177 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/delete.py", line 102, in handle
2021-05-18 12:30:10.257 177 ERROR armada.cli known_release_names = [release[0] for release in tiller.list_charts()]
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 361, in list_charts
2021-05-18 12:30:10.257 177 ERROR armada.cli for latest_release in self.list_releases():
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 252, in list_releases
2021-05-18 12:30:10.257 177 ERROR armada.cli releases = get_results()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 220, in get_results
2021-05-18 12:30:10.257 177 ERROR armada.cli for message in response:
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 364, in __next__
2021-05-18 12:30:10.257 177 ERROR armada.cli return self._next()
2021-05-18 12:30:10.257 177 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next
2021-05-18 12:30:10.257 177 ERROR armada.cli raise self
2021-05-18 12:30:10.257 177 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2021-05-18 12:30:10.257 177 ERROR armada.cli status = StatusCode.UNKNOWN
2021-05-18 12:30:10.257 177 ERROR armada.cli details = "write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe"
2021-05-18 12:30:10.257 177 ERROR armada.cli debug_error_string = "{"created":"@1621341009.251408267","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"write tcp 172.16.166.171:34140->10.10.56.10:5432: write: broken pipe","grpc_status":2}"
2021-05-18 12:30:10.257 177 ERROR armada.cli >
2021-05-18 12:30:10.257 177 ERROR armada.cli
command terminated with exit code 1
controller-0:~$