Brief Description
-----------------
StarlingX installation failed during Provisioning(no Sanity test executed) on Standard baremetal RC5.0 and Master. Stx-openstack apply-failed because of:
Error while installing release osh-openstack-rabbitmq: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "release osh-openstack-rabbitmq failed: timed out waiting for the condition"
debug_error_string = "{"created":"@1621431871.635155914","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-rabbitmq failed: timed out waiting for the condition","grpc_status":2}"
Pod issue:
openstack osh-openstack-rabbitmq-cluster-wait-ht2ws 0/1 Init:0/2 0 45m
Severity
--------
<Critical: System/Feature is not usable due to the defect>
Steps to Reproduce
------------------
Install latest RC5.0 image
Expected Behavior
------------------
Stx should install fine
Actual Behavior
----------------
Stx installation failed:
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller [-] [chart=openstack-rabbitmq]: Error while installing release osh-openstack-rabbitmq: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "release osh-openstack-rabbitmq failed: timed out waiting for the condition"
debug_error_string = "{"created":"@1621431871.635155914","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-rabbitmq failed: timed out waiting for the condition","grpc_status":2}"
>
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller Traceback (most recent call last):
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 473, in install_release
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller metadata=self.metadata)
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, None)
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline)
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller status = StatusCode.UNKNOWN
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller details = "release osh-openstack-rabbitmq failed: timed out waiting for the condition"
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller debug_error_string = "{"created":"@1621431871.635155914","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-rabbitmq failed: timed out waiting for the condition","grpc_status":2}"
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller >
2021-05-19 13:44:31.635 649 ERROR armada.handlers.tiller
2021-05-19 13:44:31.636 649 DEBUG armada.handlers.tiller [-] [chart=openstack-rabbitmq]: Helm getting release status for release=osh-openstack-rabbitmq, version=0 get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:539
2021-05-19 13:44:31.803 649 DEBUG armada.handlers.tiller [-] [chart=openstack-rabbitmq]: GetReleaseStatus= name: "osh-openstack-rabbitmq"
info {
status {
code: FAILED
}
first_deployed {
seconds: 1621430071
nanos: 418993684
}
last_deployed {
seconds: 1621430071
nanos: 418993684
}
Description: "Release \"osh-openstack-rabbitmq\" failed: timed out waiting for the condition"
}
namespace: "openstack"
get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:547
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada [-] Chart deploy [openstack-rabbitmq] failed: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-openstack-rabbitmq - Tiller Message: b'Release "osh-openstack-rabbitmq" failed: timed out waiting for the condition'
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada Traceback (most recent call last):
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 473, in install_release
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada metadata=self.metadata)
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada return _end_unary_response_blocking(state, call, False, None)
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada raise _Rendezvous(state, None, None, deadline)
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada status = StatusCode.UNKNOWN
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada details = "release osh-openstack-rabbitmq failed: timed out waiting for the condition"
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada debug_error_string = "{"created":"@1621431871.635155914","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-rabbitmq failed: timed out waiting for the condition","grpc_status":2}"
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada >
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada During handling of the above exception, another exception occurred:
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada Traceback (most recent call last):
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada result = get_result()
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 236, in <lambda>
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada if (handle_result(chart, lambda: deploy_chart(chart))):
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases)
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 239, in execute
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada timeout=timer)
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 486, in install_release
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada raise ex.ReleaseException(release, status, 'Install')
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-openstack-rabbitmq - Tiller Message: b'Release "osh-openstack-rabbitmq" failed: timed out waiting for the condition'
2021-05-19 13:44:31.804 649 ERROR armada.handlers.armada
2021-05-19 13:44:31.805 649 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['openstack-rabbitmq']
2021-05-19 13:44:32.474 649 INFO armada.handlers.lock [-] Releasing lock
2021-05-19 13:44:32.480 649 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-rabbitmq']
2021-05-19 13:44:32.480 649 ERROR armada.cli Traceback (most recent call last):
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2021-05-19 13:44:32.480 649 ERROR armada.cli self.invoke()
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2021-05-19 13:44:32.480 649 ERROR armada.cli resp = self.handle(documents, tiller)
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2021-05-19 13:44:32.480 649 ERROR armada.cli return future.result()
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-05-19 13:44:32.480 649 ERROR armada.cli return self.__get_result()
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-05-19 13:44:32.480 649 ERROR armada.cli raise self._exception
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-05-19 13:44:32.480 649 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2021-05-19 13:44:32.480 649 ERROR armada.cli return armada.sync()
2021-05-19 13:44:32.480 649 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync
2021-05-19 13:44:32.480 649 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures)
2021-05-19 13:44:32.480 649 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-rabbitmq']
2021-05-19 13:44:32.480 649 ERROR armada.cli
command terminated with exit code 1
Reproducibility
---------------
I don't have yet this info
System Configuration
--------------------
Multi-node system
Branch/Pull Time/Commit
-----------------------
RC5.0 and Master
Last Pass
---------
yesterday
Timestamp/Logs
--------------
will be attached
Test Activity
-------------
Sanity
Workaround
----------
-
This issue is because the osh-openstack- rabbitmq- rabbitmq- 0 is keeping crash.
controller-0:~$ kubectl logs -n openstack osh-openstack- rabbitmq- rabbitmq- 0 rabbitmq- rabbitmq- 0
++ echo osh-openstack-
++ awk -F - '{print $NF}'
+ POD_INCREMENT=0
+ '[' 0 -eq 0 ']'
+ exec rabbitmq-server
Failed to create thread: Resource temporarily unavailable (11)
another found:
controller-0 build info is not same as controller-1 , controller-0 is based on 5.0 , but controller-1 is in master. it is very strange.