Application apply aborted at 62% on regular system load "20190724T013000Z" . Similar to above logs 2019-07-25 12:58:28.876 106112 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-placement, overall completion: 62.0% 2019-07-25 13:28:27.382 26231 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:173 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller [-] [chart=openstack-libvirt]: Error while installing release osh-openstack-libvirt: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.UNKNOWN details = "release osh-openstack-libvirt failed: timed out waiting for the condition" debug_error_string = "{"created":"@1564061309.128669855","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-libvirt failed: timed out waiting for the condition","grpc_status":2}" > 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller Traceback (most recent call last): 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 465, in install_release 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller metadata=self.metadata) 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__ 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, None) 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline) 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller status = StatusCode.UNKNOWN 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller details = "release osh-openstack-libvirt failed: timed out waiting for the condition" 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller debug_error_string = "{"created":"@1564061309.128669855","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-libvirt failed: timed out waiting for the condition","grpc_status":2}" 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller > 2019-07-25 13:28:29.128 26231 ERROR armada.handlers.tiller 2019-07-25 13:28:29.130 26231 DEBUG armada.handlers.tiller [-] [chart=openstack-libvirt]: Helm getting release status for release=osh-openstack-libvirt, version=0 get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:531 2019-07-25 13:28:29.219 26231 DEBUG armada.handlers.tiller [-] [chart=openstack-libvirt]: GetReleaseStatus= name: "osh-openstack-libvirt" info { status { code: FAILED } first_deployed { seconds: 1564059509 nanos: 495775 } last_deployed { seconds: 1564059509 nanos: 495775 } Description: "Release \"osh-openstack-libvirt\" failed: timed out waiting for the condition" } namespace: "openstack" get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:539 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada [-] Chart deploy [openstack-libvirt] failed: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-openstack-libvirt - Tiller Message: b'Release "osh-openstack-libvirt" failed: timed out waiting for the condition' 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada Traceback (most recent call last): 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 465, in install_release 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada metadata=self.metadata) 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__ 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada return _end_unary_response_blocking(state, call, False, None) 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada raise _Rendezvous(state, None, None, deadline) 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada status = StatusCode.UNKNOWN 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada details = "release osh-openstack-libvirt failed: timed out waiting for the condition" 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada debug_error_string = "{"created":"@1564061309.128669855","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-libvirt failed: timed out waiting for the condition","grpc_status":2}" 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada > 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada During handling of the above exception, another exception occurred: 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada Traceback (most recent call last): 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 223, in handle_result 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada result = get_result() 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada return self.__get_result() 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada raise self._exception 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada result = self.fn(*self.args, **self.kwargs) 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 212, in deploy_chart 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada prefix, known_releases) 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 234, in execute 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada timeout=timer) 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 478, in install_release 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada raise ex.ReleaseException(release, status, 'Install') 2019-07-25 13:28:29.220 26231 ERROR armada.handlers.armada armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-openstack-libvirt - Tiller Message: b'Release "osh-openstack-libvirt" failed: timed out waiting for the condition' 2019-07-25 13:28:29.220 26231