platform-integ-apps apply-failed since install with InvalidOverrideFileException

[Yang Liu]

Brief Description
-----------------
platform-integ-apps in apply-failed state after controller-0 unlock with InvalidOverrideFileException in armada log.

2020-06-15 22:05:55.886 56 ERROR armada.cli [-] Caught internal exception: armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml is not a valid override file.
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 37, in _load_yaml_file
2020-06-15 22:05:55.886 56 ERROR armada.cli with open(doc) as f:
2020-06-15 22:05:55.886 56 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml'

Manual reapply fails with same error.

Severity
--------
Major

Steps to Reproduce
------------------
Install and configure an AIO-SX system
Check system application-list

Expected Behavior
------------------
system applications are applied or uploaded

Actual Behavior
----------------
platform-integ-apps is apply-failed

Reproducibility
---------------
Likely intermittent because same load passed on other sx system.

System Configuration
--------------------
One node system
Lab-name: ml350-g10-1

Branch/Pull Time/Commit
-----------------------
20200614 load

Last Pass
---------
- Same load on a different sx system (wcp112) passed.
- Same system with 20200612 load passed

Timestamp/Logs
--------------
Seems multiple apply were attempted automatically, and all of them failed with same exception:

-rw-r--r-- 1 1000 users 4812 Jun 15 22:05 platform-integ-apps-apply_2020-06-15-22-05-54.log
-rw-r--r-- 1 1000 users 4812 Jun 15 22:06 platform-integ-apps-apply_2020-06-15-22-06-26.log
-rw-r--r-- 1 1000 users 4812 Jun 15 22:06 platform-integ-apps-apply_2020-06-15-22-06-57.log
-rw-r--r-- 1 1000 users 4850 Jun 15 22:07 platform-integ-apps-apply_2020-06-15-22-07-28.log
-rw-r--r-- 1 1000 users 4850 Jun 15 22:08 platform-integ-apps-apply_2020-06-15-22-08-00.log

2020-06-15 22:05:55.886 56 ERROR armada.cli [-] Caught internal exception: armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml is not a valid override file.
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 37, in _load_yaml_file
2020-06-15 22:05:55.886 56 ERROR armada.cli with open(doc) as f:
2020-06-15 22:05:55.886 56 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml'
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli During handling of the above exception, another exception occurred:
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2020-06-15 22:05:55.886 56 ERROR armada.cli self.invoke()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2020-06-15 22:05:55.886 56 ERROR armada.cli resp = self.handle(documents, tiller)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2020-06-15 22:05:55.886 56 ERROR armada.cli return future.result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2020-06-15 22:05:55.886 56 ERROR armada.cli return self.__get_result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2020-06-15 22:05:55.886 56 ERROR armada.cli raise self._exception
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2020-06-15 22:05:55.886 56 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 255, in handle
2020-06-15 22:05:55.886 56 ERROR armada.cli target_manifest=self.target_manifest)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 87, in __init__
2020-06-15 22:05:55.886 56 ERROR armada.cli values=values).update_manifests()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 146, in update_manifests
2020-06-15 22:05:55.886 56 ERROR armada.cli merging_values = self._load_yaml_file(value)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 40, in _load_yaml_file
2020-06-15 22:05:55.886 56 ERROR armada.cli raise override_exceptions.InvalidOverrideFileException(doc)
2020-06-15 22:05:55.886 56 ERROR armada.cli armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml is not a valid override file.
2020-06-15 22:05:55.886 56 ERROR armada.cli ^[[00m

Test Activity
-------------
Install and Config

[Yang Liu (yliu12)]

Logs: https://files.starlingx.kube.cengn.ca/launchpad/1883613

[Ghada Khalil (gkhalil)]

One time occurrence in one lab. Request that the reporter monitor for a re-occurrence.

[Ghada Khalil (gkhalil)]

Agreed with Yang to close for now since the issue is not reproducible. She will monitor and re-open/re-gate if required.