Comment 10 for bug 1841435

Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

The workaround listed above does not seem to help.
Still failed on reapply

a. Deployment manager continues to attempt to unlock controller-1
2019-08-29 18:18:24.700 112975 INFO sysinv.api.controllers.v1.host [-] controller-1 ihost check_unlock_application
2019-08-29 18:18:24.703 112975 INFO sysinv.api.controllers.v1.host [-] host unlock check didn't pass, so set the ihost_action back to None and re-raise the exception
2019-08-29 18:18:24.713 112975 WARNING wsme.api [-] Client-side error: Rejected: Can not unlock host controller-1 while an application is being applied, updated or recovered. Please try again later.

ceph -s shows ceph is health_ok

platform-integ-apps-apply.log
2019-08-29 18:18:02.612 7736 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:173
2019-08-29 18:19:02.681 7736 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:173
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller [-] [chart=kube-system-rbd-provisioner]: Error while installing release stx-rbd-provisioner: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "release stx-rbd-provisioner failed: timed out waiting for the condition"
        debug_error_string = "{"created":"@1567102801.090806895","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release stx-rbd-provisioner failed: timed out waiting for the condition","grpc_status":2}"
>
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller Traceback (most recent call last):
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 465, in install_release
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller metadata=self.metadata)
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, None)
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline)
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller status = StatusCode.UNKNOWN
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller details = "release stx-rbd-provisioner failed: timed out waiting for the condition"
2019-08-29 18:20:01.091 7736 ERROR armada.handlers.tiller debug_error_string = "{"created":"@1567102801.090806895","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release stx-rbd-provisioner failed: timed out waiting for the condition","grpc_status":2}"