The workaround listed above does not seem to help.
Still failed on reapply
a. Deployment manager continues to attempt to unlock controller-1
2019-08-29 18:18:24.700 112975 INFO sysinv.api.controllers.v1.host [-] controller-1 ihost check_unlock_application
2019-08-29 18:18:24.703 112975 INFO sysinv.api.controllers.v1.host [-] host unlock check didn't pass, so set the ihost_action back to None and re-raise the exception
2019-08-29 18:18:24.713 112975 WARNING wsme.api [-] Client-side error: Rejected: Can not unlock host controller-1 while an application is being applied, updated or recovered. Please try again later.
ceph -s shows ceph is health_ok
platform-integ-apps-apply.log
2019-08-29 18:18:02.612 7736 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:173
2019-08-29 18:19:02.681 7736 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:173
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller [-] [chart=kube-system-rbd-provisioner]: Error while installing release stx-rbd-provisioner: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "release stx-rbd-provisioner failed: timed out waiting for the condition" debug_error_string = "{"created":"@1567102801.090806895","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release stx-rbd-provisioner failed: timed out waiting for the condition","grpc_status":2}"
>
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller Traceback (most recent call last):
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 465, in install_release
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller metadata=self.metadata)
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, None)
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline)
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller status = StatusCode.UNKNOWN
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller details = "release stx-rbd-provisioner failed: timed out waiting for the condition"
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller debug_error_string = "{"created":"@1567102801.090806895","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release stx-rbd-provisioner failed: timed out waiting for the condition","grpc_status":2}"
The workaround listed above does not seem to help.
Still failed on reapply
a. Deployment manager continues to attempt to unlock controller-1 api.controllers .v1.host [-] controller-1 ihost check_unlock_ application api.controllers .v1.host [-] host unlock check didn't pass, so set the ihost_action back to None and re-raise the exception
2019-08-29 18:18:24.700 112975 INFO sysinv.
2019-08-29 18:18:24.703 112975 INFO sysinv.
2019-08-29 18:18:24.713 112975 WARNING wsme.api [-] Client-side error: Rejected: Can not unlock host controller-1 while an application is being applied, updated or recovered. Please try again later.
ceph -s shows ceph is health_ok
platform- integ-apps- apply.log handlers. lock [-] Updating lock update_lock /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ lock.py: 173 handlers. lock [-] Updating lock update_lock /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ lock.py: 173 handlers. tiller [-] [chart= kube-system- rbd-provisioner ]: Error while installing release stx-rbd- provisioner: grpc._channel. _Rendezvous: <_Rendezvous of RPC that terminated with:
debug_ error_string = "{"created" :"@1567102801. 090806895" ,"description" :"Error received from peer"," file":" src/core/ lib/surface/ call.cc" ,"file_ line":1017, "grpc_message" :"release stx-rbd-provisioner failed: timed out waiting for the condition" ,"grpc_ status" :2}" handlers. tiller Traceback (most recent call last): handlers. tiller File "/usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ tiller. py", line 465, in install_release handlers. tiller metadata= self.metadata) handlers. tiller File "/usr/local/ lib/python3. 6/dist- packages/ grpc/_channel. py", line 533, in __call__ handlers. tiller return _end_unary_ response_ blocking( state, call, False, None) handlers. tiller File "/usr/local/ lib/python3. 6/dist- packages/ grpc/_channel. py", line 467, in _end_unary_ response_ blocking handlers. tiller raise _Rendezvous(state, None, None, deadline) handlers. tiller grpc._channel. _Rendezvous: <_Rendezvous of RPC that terminated with: handlers. tiller status = StatusCode.UNKNOWN handlers. tiller details = "release stx-rbd-provisioner failed: timed out waiting for the condition" handlers. tiller debug_error_string = "{"created" :"@1567102801. 090806895" ,"description" :"Error received from peer"," file":" src/core/ lib/surface/ call.cc" ,"file_ line":1017, "grpc_message" :"release stx-rbd-provisioner failed: timed out waiting for the condition" ,"grpc_ status" :2}"
2019-08-29 18:18:02.612 7736 DEBUG armada.
2019-08-29 18:19:02.681 7736 DEBUG armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
status = StatusCode.UNKNOWN
details = "release stx-rbd-provisioner failed: timed out waiting for the condition"
>
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.
2019-08-29 18:20:01.091 7736 ERROR armada.