platform-integ-apps fail to apply initially due to could not find the requested resource (get configmaps)

[Peng Peng]

Brief Description
-----------------
After initial install and config, platform-integ-apps in apply-failed status due to "the server could not find the requested resource (get configmaps)"

Severity
--------
minor

Steps to Reproduce
------------------
Install and configure a stx system

Expected Behavior
------------------
- platform-integ-apps automatically applied successfully

Actual Behavior
----------------
- platform-integ-apps occasionally in apply-failed

Reproducibility
---------------
Unknown - first time this is seen it
looks similar with LP-1850189

System Configuration
--------------------
One node system

Lab-name: SM-4

Branch/Pull Time/Commit
-----------------------
20200111T023000Z

Last Pass
---------
20200107T000000Z

Timestamp/Logs
--------------
system application-list
+---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail |
+---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+

2020-01-14 07:57:24.225 107 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "the server could not find the requested resource (get configmaps)"
        debug_error_string = "{"created":"@1578988643.425059681","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"the server could not find the requested resource (get configmaps)","grpc_status":2}"
>
2020-01-14 07:57:24.225 107 ERROR armada.cli Traceback (most recent call last):
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2020-01-14 07:57:24.225 107 ERROR armada.cli self.invoke()
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2020-01-14 07:57:24.225 107 ERROR armada.cli resp = self.handle(documents, tiller)
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2020-01-14 07:57:24.225 107 ERROR armada.cli return future.result()
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2020-01-14 07:57:24.225 107 ERROR armada.cli return self.__get_result()
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2020-01-14 07:57:24.225 107 ERROR armada.cli raise self._exception
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2020-01-14 07:57:24.225 107 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2020-01-14 07:57:24.225 107 ERROR armada.cli return armada.sync()
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 189, in sync
2020-01-14 07:57:24.225 107 ERROR armada.cli known_releases = self.tiller.list_releases()
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 252, in list_releases
2020-01-14 07:57:24.225 107 ERROR armada.cli releases = get_results()
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 220, in get_results
2020-01-14 07:57:24.225 107 ERROR armada.cli for message in response:
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 364, in __next__
2020-01-14 07:57:24.225 107 ERROR armada.cli return self._next()
2020-01-14 07:57:24.225 107 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next
2020-01-14 07:57:24.225 107 ERROR armada.cli raise self
2020-01-14 07:57:24.225 107 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2020-01-14 07:57:24.225 107 ERROR armada.cli status = StatusCode.UNKNOWN
2020-01-14 07:57:24.225 107 ERROR armada.cli details = "the server could not find the requested resource (get configmaps)"
2020-01-14 07:57:24.225 107 ERROR armada.cli debug_error_string = "{"created":"@1578988643.425059681","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"the server could not find the requested resource (get configmaps)","grpc_status":2}"

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

logs @
https://files.starlingx.kube.cengn.ca/launchpad/1859645

[Anujeyan Manokeran (anujeyan)]

This issue was seen in SM-2 as well on load 2020-01-17 02:30:05.

[Ghada Khalil (gkhalil)]

stx.4.0 / high priority - issue seen twice recently. According to Yang, a re-apply doesn't work in this case.

[Peng Peng (ppeng)]

Issue reproduced on
Lab: WCP_71_75
Load: 2020-01-22_20-00-00
log @
https://files.starlingx.kube.cengn.ca/launchpad/1859645

[Peng Peng (ppeng)]

Issue reproduced on
Lab: WCP_71_75
Load: 2020-01-28_18-49-15
log @
https://files.starlingx.kube.cengn.ca/launchpad/1859645

[Ghada Khalil (gkhalil)]

Bob Church reviewed the last occurrence and confirmed that this is a duplicate of https://bugs.launchpad.net/starlingx/+bug/1856078

Tiller does not have the appropriate environment variables set to connect to the k8s api server.

If this is encountered again, you can confirm that it's the same issue by running:
helm ls -a

The command should return:
Error: the server could not find the requested resource (get configmaps)

The workaround is to restart the tiller pod using the following command:
kubectl get pods --all-namespaces -o wide | grep tiller-deploy
kubectl delete pods -n kube-system <tiller-deploy-...>

[Ghada Khalil (gkhalil)]

Updating the status to match the duplicate LP: https://bugs.launchpad.net/starlingx/+bug/1856078
Merged on 2020-04-22