Comment 16 for bug 1856078

Issue was apparent on clean install
Lab (IPV6): WP-3-7
2020-02-22_11-13-25

see sysinv.log (controller-0 log attached)

sysinv 2020-02-24 16:02:15.565 274704 INFO sysinv.conductor.kube_app [-] Armada service started!
sysinv 2020-02-24 16:02:15.566 274704 INFO sysinv.conductor.kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable-chart-cleanup --debug /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-ceph-pools-audit.yaml --values /overrides/platform-integ-apps/1.0-8/helm-toolkit-helm-toolkit.yaml --tiller-host tiller-deploy.kube-system.svc.cluster.local | tee /logs/platform-integ-apps-apply_2020-02-24-16-02-15.log'
sysinv 2020-02-24 16:02:16.376 274704 INFO sysinv.conductor.kube_app [-] Starting progress monitoring thread for app platform-integ-apps
sysinv 2020-02-24 16:02:17.513 274704 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml. See /var/log/armada/platform-integ-apps-apply_2020-02-24-16-02-15.log for details.

see also armada log.

 get_results /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:215
2020-02-24 16:02:17.327 16 INFO armada.handlers.lock [-] Releasing lock
2020-02-24 16:02:17.332 16 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
 status = StatusCode.UNKNOWN
 details = "the server could not find the requested resource (get configmaps)"
 debug_error_string = "{"created":"@1582560136.998050126","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"the server could not find the requested resource (get configmaps)","grpc_status":2}"
>

2020-02-24 16:02:17.332 16 ERROR armada.cli Traceback (most recent call last):
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke

2020-02-24 16:02:17.332 16 ERROR armada.cli self.invoke()
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2020-02-24 16:02:17.332 16 ERROR armada.cli resp = self.handle(documents, tiller)
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2020-02-24 16:02:17.332 16 ERROR armada.cli return future.result()
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2020-02-24 16:02:17.332 16 ERROR armada.cli return self.__get_result()

2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2020-02-24 16:02:17.332 16 ERROR armada.cli raise self._exception

2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2020-02-24 16:02:17.332 16 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)

2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2020-02-24 16:02:17.332 16 ERROR armada.cli return armada.sync()
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 189, in sync

2020-02-24 16:02:17.332 16 ERROR armada.cli known_releases = self.tiller.list_releases()
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 252, in list_releases
2020-02-24 16:02:17.332 16 ERROR armada.cli releases = get_results()
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 220, in get_results
2020-02-24 16:02:17.332 16 ERROR armada.cli for message in response:
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 364, in __next__

2020-02-24 16:02:17.332 16 ERROR armada.cli return self._next()
2020-02-24 16:02:17.332 16 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next
2020-02-24 16:02:17.332 16 ERROR armada.cli raise self

2020-02-24 16:02:17.332 16 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2020-02-24 16:02:17.332 16 ERROR armada.cli status = StatusCode.UNKNOWN
2020-02-24 16:02:17.332 16 ERROR armada.cli details = "the server could not find the requested resource (get configmaps)"
2020-02-24 16:02:17.332 16 ERROR armada.cli debug_error_string = "{"created":"@1582560136.998050126","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"the server could not find the requested resource (get configmaps)","grpc_status":2}"