Same error we see on the MONOLITHIC "20200701T054008Z" build.
2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller [-] [chart=openstack-mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.UNKNOWN details = "release osh-openstack-mariadb failed: timed out waiting for the condition" debug_error_string = "{"created":"@1593614153.871875172","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" > 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller Traceback (most recent call last): 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 473, in install_release 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller metadata=self.metadata) 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__ 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, None) 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline) 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller status = StatusCode.UNKNOWN 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller details = "release osh-openstack-mariadb failed: timed out waiting for the condition" 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller debug_error_string = "{"created":"@1593614153.871875172","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller > 2020-07-01 14:35:53.872 550 ERROR armada.handlers.tiller 2020-07-01 14:35:53.873 550 DEBUG armada.handlers.tiller [-] [chart=openstack-mariadb]: Helm getting release status for release=osh-openstack-mariadb, version=0 get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:539 2020-07-01 14:35:54.029 550 DEBUG armada.handlers.tiller [-] [chart=openstack-mariadb]: GetReleaseStatus= name: "osh-openstack-mariadb" info { status { code: FAILED } first_deployed { seconds: 1593612353 nanos: 337681515 } last_deployed { seconds: 1593612353 nanos: 337681515 } Description: "Release \"osh-openstack-mariadb\" failed: timed out waiting for the condition" } namespace: "openstack" get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:547 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada [-] Chart deploy [openstack-mariadb] failed: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-openstack-mariadb - Tiller Message: b'Release "osh-openstack-mariadb" failed: timed out waiting for the condition' 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada Traceback (most recent call last): 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 473, in install_release 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada metadata=self.metadata) 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__ 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada return _end_unary_response_blocking(state, call, False, None) 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada raise _Rendezvous(state, None, None, deadline) 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada status = StatusCode.UNKNOWN 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada details = "release osh-openstack-mariadb failed: timed out waiting for the condition" 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada debug_error_string = "{"created":"@1593614153.871875172","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada > 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada During handling of the above exception, another exception occurred: 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada Traceback (most recent call last): 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada result = get_result() 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 236, in <lambda> 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada if (handle_result(chart, lambda: deploy_chart(chart))): 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases) 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 239, in execute 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada timeout=timer) 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 486, in install_release 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada raise ex.ReleaseException(release, status, 'Install') 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-openstack-mariadb - Tiller Message: b'Release "osh-openstack-mariadb" failed: timed out waiting for the condition' 2020-07-01 14:35:54.030 550 ERROR armada.handlers.armada 2020-07-01 14:35:54.031 550 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['openstack-mariadb'] 2020-07-01 14:35:54.252 550 INFO armada.handlers.lock [-] Releasing lock 2020-07-01 14:35:54.256 550 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-mariadb'] 2020-07-01 14:35:54.256 550 ERROR armada.cli Traceback (most recent call last): 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke 2020-07-01 14:35:54.256 550 ERROR armada.cli self.invoke() 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke 2020-07-01 14:35:54.256 550 ERROR armada.cli resp = self.handle(documents, tiller) 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper 2020-07-01 14:35:54.256 550 ERROR armada.cli return future.result() 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result 2020-07-01 14:35:54.256 550 ERROR armada.cli return self.__get_result() 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2020-07-01 14:35:54.256 550 ERROR armada.cli raise self._exception 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2020-07-01 14:35:54.256 550 ERROR armada.cli result = self.fn(*self.args, **self.kwargs) 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle 2020-07-01 14:35:54.256 550 ERROR armada.cli return armada.sync() 2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync 2020-07-01 14:35:54.256 550 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures) 2020-07-01 14:35:54.256 550 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-mariadb'] 2020-07-01 14:35:54.256 550 ERROR armada.cli
Same error we see on the MONOLITHIC "20200701T054008Z" build.
2020-07-01 14:35:53.872 550 ERROR armada. handlers. tiller [-] [chart= openstack- mariadb] : Error while installing release osh-openstack- mariadb: grpc._channel. _Rendezvous: <_Rendezvous of RPC that terminated with: mariadb failed: timed out waiting for the condition"
debug_ error_string = "{"created" :"@1593614153. 871875172" ,"description" :"Error received from peer"," file":" src/core/ lib/surface/ call.cc" ,"file_ line":1017, "grpc_message" :"release osh-openstack- mariadb failed: timed out waiting for the condition" ,"grpc_ status" :2}" handlers. tiller Traceback (most recent call last): handlers. tiller File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py", line 473, in install_release handlers. tiller metadata= self.metadata) handlers. tiller File "/usr/local/ lib/python3. 6/dist- packages/ grpc/_channel. py", line 533, in __call__ handlers. tiller return _end_unary_ response_ blocking( state, call, False, None) handlers. tiller File "/usr/local/ lib/python3. 6/dist- packages/ grpc/_channel. py", line 467, in _end_unary_ response_ blocking handlers. tiller raise _Rendezvous(state, None, None, deadline) handlers. tiller grpc._channel. _Rendezvous: <_Rendezvous of RPC that terminated with: handlers. tiller status = StatusCode.UNKNOWN handlers. tiller details = "release osh-openstack- mariadb failed: timed out waiting for the condition" handlers. tiller debug_error_string = "{"created" :"@1593614153. 871875172" ,"description" :"Error received from peer"," file":" src/core/ lib/surface/ call.cc" ,"file_ line":1017, "grpc_message" :"release osh-openstack- mariadb failed: timed out waiting for the condition" ,"grpc_ status" :2}" handlers. tiller > handlers. tiller handlers. tiller [-] [chart= openstack- mariadb] : Helm getting release status for release= osh-openstack- mariadb, version=0 get_release_status /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:539 handlers. tiller [-] [chart= openstack- mariadb] : GetReleaseStatus= name: "osh-openstack- mariadb" -mariadb\ " failed: timed out waiting for the condition" lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py:547 handlers. armada [-] Chart deploy [openstack-mariadb] failed: armada. exceptions. tiller_ exceptions. ReleaseExceptio n: Failed to Install release: osh-openstack- mariadb - Tiller Message: b'Release "osh-openstack- mariadb" failed: timed out waiting for the condition' handlers. armada Traceback (most recent call last): handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py", line 473, in install_release handlers. armada metadata= self.metadata) handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ grpc/_channel. py", line 533, in __call__ handlers. armada return _end_unary_ response_ blocking( state, call, False, None) handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ grpc/_channel. py", line 467, in _end_unary_ response_ blocking handlers. armada raise _Rendezvous(state, None, None, deadline) handlers. armada grpc._channel. _Rendezvous: <_Rendezvous of RPC that terminated with: handlers. armada status = StatusCode.UNKNOWN handlers. armada details = "release osh-openstack- mariadb failed: timed out waiting for the condition" handlers. armada debug_error_string = "{"created" :"@1593614153. 871875172" ,"description" :"Error received from peer"," file":" src/core/ lib/surface/ call.cc" ,"file_ line":1017, "grpc_message" :"release osh-openstack- mariadb failed: timed out waiting for the condition" ,"grpc_ status" :2}" handlers. armada > handlers. armada handlers. armada During handling of the above exception, another exception occurred: handlers. armada handlers. armada Traceback (most recent call last): handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ armada. py", line 225, in handle_result handlers. armada result = get_result() handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ armada. py", line 236, in <lambda> handlers. armada if (handle_ result( chart, lambda: deploy_ chart(chart) )): handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ armada. py", line 214, in deploy_chart handlers. armada chart, cg_test_all_charts, prefix, known_releases) handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ chart_deploy. py", line 239, in execute handlers. armada timeout=timer) handlers. armada File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py", line 486, in install_release handlers. armada raise ex.ReleaseExcep tion(release, status, 'Install') handlers. armada armada. exceptions. tiller_ exceptions. ReleaseExceptio n: Failed to Install release: osh-openstack- mariadb - Tiller Message: b'Release "osh-openstack- mariadb" failed: timed out waiting for the condition' handlers. armada handlers. armada [-] Chart deploy(s) failed: ['openstack- mariadb' ] handlers. lock [-] Releasing lock exceptions. armada_ exceptions. ChartDeployExce ption: Exception deploying charts: ['openstack- mariadb' ] lib/python3. 6/dist- packages/ armada/ cli/__init_ _.py", line 38, in safe_invoke lib/python3. 6/dist- packages/ armada/ cli/apply. py", line 213, in invoke documents, tiller) lib/python3. 6/dist- packages/ armada/ handlers/ lock.py" , line 81, in func_wrapper python3. 6/concurrent/ futures/ _base.py" , line 425, in result python3. 6/concurrent/ futures/ _base.py" , line 384, in __get_result python3. 6/concurrent/ futures/ thread. py", line 56, in run lib/python3. 6/dist- packages/ armada/ cli/apply. py", line 256, in handle lib/python3. 6/dist- packages/ armada/ handlers/ armada. py", line 252, in sync exceptions. ChartDeployExce ption(failures) exceptions. armada_ exceptions. ChartDeployExce ption: Exception deploying charts: ['openstack- mariadb' ]
status = StatusCode.UNKNOWN
details = "release osh-openstack-
>
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.872 550 ERROR armada.
2020-07-01 14:35:53.873 550 DEBUG armada.
2020-07-01 14:35:54.029 550 DEBUG armada.
info {
status {
code: FAILED
}
first_deployed {
seconds: 1593612353
nanos: 337681515
}
last_deployed {
seconds: 1593612353
nanos: 337681515
}
Description: "Release \"osh-openstack
}
namespace: "openstack"
get_release_status /usr/local/
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.030 550 ERROR armada.
2020-07-01 14:35:54.031 550 ERROR armada.
2020-07-01 14:35:54.252 550 INFO armada.
2020-07-01 14:35:54.256 550 ERROR armada.cli [-] Caught internal exception: armada.
2020-07-01 14:35:54.256 550 ERROR armada.cli Traceback (most recent call last):
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/
2020-07-01 14:35:54.256 550 ERROR armada.cli self.invoke()
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/
2020-07-01 14:35:54.256 550 ERROR armada.cli resp = self.handle(
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/
2020-07-01 14:35:54.256 550 ERROR armada.cli return future.result()
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/lib/
2020-07-01 14:35:54.256 550 ERROR armada.cli return self.__get_result()
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/lib/
2020-07-01 14:35:54.256 550 ERROR armada.cli raise self._exception
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/lib/
2020-07-01 14:35:54.256 550 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/
2020-07-01 14:35:54.256 550 ERROR armada.cli return armada.sync()
2020-07-01 14:35:54.256 550 ERROR armada.cli File "/usr/local/
2020-07-01 14:35:54.256 550 ERROR armada.cli raise armada_
2020-07-01 14:35:54.256 550 ERROR armada.cli armada.
2020-07-01 14:35:54.256 550 ERROR armada.cli