Comment 6 for bug 1849688

Hit this issue again after lock/unlock on the single node system
2019-10-23_20-00-00

$ system application-list
...
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail

2019-10-24 20:09:03.486 11 ERROR armada.handlers.wait [-] [chart=kube-system-rbd-provisioner]: Timed out waiting for pods (namespace=kube-system, labels=(app=rbd-provisioner)). These pods were not ready=['rbd-provisioner-7484d49cf6-dfx6h']
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada [-] Chart deploy [kube-system-rbd-provisioner] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=kube-system, labels=(app=rbd-provisioner)). These pods were not ready=['rbd-provisioner-7484d49cf6-dfx6h']
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada Traceback (most recent call last):
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada result = get_result()
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 236, in <lambda>
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada if (handle_result(chart, lambda: deploy_chart(chart))):
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases)
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 248, in execute
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada chart_wait.wait(timer)
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, in wait
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada wait.wait(timeout=timeout)
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, in wait
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada modified = self._wait(deadline)
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, in _wait
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada raise k8s_exceptions.KubernetesWatchTimeoutException(error)
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=kube-system, labels=(app=rbd-provisioner)). These pods were not ready=['rbd-provisioner-7484d49cf6-dfx6h']
2019-10-24 20:09:03.487 11 ERROR armada.handlers.armada
2019-10-24 20:09:03.489 11 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['kube-system-rbd-provisioner']
2019-10-24 20:09:03.979 11 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['kube-system-rbd-provisioner']
2019-10-24 20:09:03.979 11 ERROR armada.cli Traceback (most recent call last):
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2019-10-24 20:09:03.979 11 ERROR armada.cli self.invoke()
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2019-10-24 20:09:03.979 11 ERROR armada.cli resp = self.handle(documents, tiller)
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2019-10-24 20:09:03.979 11 ERROR armada.cli return future.result()
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2019-10-24 20:09:03.979 11 ERROR armada.cli return self.__get_result()
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2019-10-24 20:09:03.979 11 ERROR armada.cli raise self._exception
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2019-10-24 20:09:03.979 11 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2019-10-24 20:09:03.979 11 ERROR armada.cli return armada.sync()
2019-10-24 20:09:03.979 11 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync
2019-10-24 20:09:03.979 11 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures)
2019-10-24 20:09:03.979 11 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['kube-system-rbd-provisioner']
2019-10-24 20:09:03.979 11 ERROR armada.cli