Brief Description
-----------------
platform-integ-apps in apply-failed state after controller-0 unlock with InvalidOverrideFileException in armada log.
2020-06-15 22:05:55.886 56 ERROR armada.cli [-] Caught internal exception: armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml is not a valid override file.
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 37, in _load_yaml_file
2020-06-15 22:05:55.886 56 ERROR armada.cli with open(doc) as f:
2020-06-15 22:05:55.886 56 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml'
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli During handling of the above exception, another exception occurred:
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2020-06-15 22:05:55.886 56 ERROR armada.cli self.invoke()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2020-06-15 22:05:55.886 56 ERROR armada.cli resp = self.handle(documents, tiller)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2020-06-15 22:05:55.886 56 ERROR armada.cli return future.result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2020-06-15 22:05:55.886 56 ERROR armada.cli return self.__get_result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2020-06-15 22:05:55.886 56 ERROR armada.cli raise self._exception
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2020-06-15 22:05:55.886 56 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 255, in handle
2020-06-15 22:05:55.886 56 ERROR armada.cli target_manifest=self.target_manifest)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 87, in __init__
2020-06-15 22:05:55.886 56 ERROR armada.cli values=values).update_manifests()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 146, in update_manifests
2020-06-15 22:05:55.886 56 ERROR armada.cli merging_values = self._load_yaml_file(value)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 40, in _load_yaml_file
2020-06-15 22:05:55.886 56 ERROR armada.cli raise override_exceptions.InvalidOverrideFileException(doc)
2020-06-15 22:05:55.886 56 ERROR armada.cli armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml is not a valid override file.
Severity
--------
Major
Steps to Reproduce
------------------
Install and configure an AIO-SX system
Check system application-list
Expected Behavior
------------------
system applications are applied or uploaded
Actual Behavior
----------------
platform-integ-apps is apply-failed
Reproducibility
---------------
Likely intermittent because same load passed on other sx system.
System Configuration
--------------------
One node system
Lab-name: ml350-g10-1
Brief Description FileException in armada log.
-----------------
platform-integ-apps in apply-failed state after controller-0 unlock with InvalidOverride
2020-06-15 22:05:55.886 56 ERROR armada.cli [-] Caught internal exception: armada. exceptions. override_ exceptions. InvalidOverride FileException: /overrides/ platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml is not a valid override file. lib/python3. 6/dist- packages/ armada/ handlers/ override. py", line 37, in _load_yaml_file platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml' lib/python3. 6/dist- packages/ armada/ cli/__init_ _.py", line 38, in safe_invoke lib/python3. 6/dist- packages/ armada/ cli/apply. py", line 213, in invoke documents, tiller) lib/python3. 6/dist- packages/ armada/ handlers/ lock.py" , line 81, in func_wrapper python3. 6/concurrent/ futures/ _base.py" , line 425, in result python3. 6/concurrent/ futures/ _base.py" , line 384, in __get_result python3. 6/concurrent/ futures/ thread. py", line 56, in run lib/python3. 6/dist- packages/ armada/ cli/apply. py", line 255, in handle manifest= self.target_ manifest) lib/python3. 6/dist- packages/ armada/ handlers/ armada. py", line 87, in __init__ values) .update_ manifests( ) lib/python3. 6/dist- packages/ armada/ handlers/ override. py", line 146, in update_manifests yaml_file( value) lib/python3. 6/dist- packages/ armada/ handlers/ override. py", line 40, in _load_yaml_file exceptions. InvalidOverride FileException( doc) exceptions. override_ exceptions. InvalidOverride FileException: /overrides/ platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml is not a valid override file.
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli with open(doc) as f:
2020-06-15 22:05:55.886 56 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli During handling of the above exception, another exception occurred:
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli self.invoke()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli resp = self.handle(
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli return future.result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/
2020-06-15 22:05:55.886 56 ERROR armada.cli return self.__get_result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/
2020-06-15 22:05:55.886 56 ERROR armada.cli raise self._exception
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/
2020-06-15 22:05:55.886 56 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli target_
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli values=
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli merging_values = self._load_
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli raise override_
2020-06-15 22:05:55.886 56 ERROR armada.cli armada.
Severity
--------
Major
Steps to Reproduce
------------------
Install and configure an AIO-SX system
Check system application-list
Expected Behavior
------------------
system applications are applied or uploaded
Actual Behavior
----------------
platform-integ-apps is apply-failed
Reproducibility
---------------
Likely intermittent because same load passed on other sx system.
System Configuration ------- ------
-------
One node system
Lab-name: ml350-g10-1
Branch/Pull Time/Commit ------- ------- --
-------
20200614 load
Last Pass
---------
- Same load on a different sx system (wcp112) passed.
- Same system with 20200612 load passed
Timestamp/Logs
--------------
Seems multiple apply were attempted automatically, and all of them failed with same exception:
-rw-r--r-- 1 1000 users 4812 Jun 15 22:05 platform- integ-apps- apply_2020- 06-15-22- 05-54.log integ-apps- apply_2020- 06-15-22- 06-26.log integ-apps- apply_2020- 06-15-22- 06-57.log integ-apps- apply_2020- 06-15-22- 07-28.log integ-apps- apply_2020- 06-15-22- 08-00.log
-rw-r--r-- 1 1000 users 4812 Jun 15 22:06 platform-
-rw-r--r-- 1 1000 users 4812 Jun 15 22:06 platform-
-rw-r--r-- 1 1000 users 4850 Jun 15 22:07 platform-
-rw-r--r-- 1 1000 users 4850 Jun 15 22:08 platform-
2020-06-15 22:05:55.886 56 ERROR armada.cli [-] Caught internal exception: armada. exceptions. override_ exceptions. InvalidOverride FileException: /overrides/ platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml is not a valid override file. lib/python3. 6/dist- packages/ armada/ handlers/ override. py", line 37, in _load_yaml_file platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml' lib/python3. 6/dist- packages/ armada/ cli/__init_ _.py", line 38, in safe_invoke lib/python3. 6/dist- packages/ armada/ cli/apply. py", line 213, in invoke documents, tiller) lib/python3. 6/dist- packages/ armada/ handlers/ lock.py" , line 81, in func_wrapper python3. 6/concurrent/ futures/ _base.py" , line 425, in result python3. 6/concurrent/ futures/ _base.py" , line 384, in __get_result python3. 6/concurrent/ futures/ thread. py", line 56, in run lib/python3. 6/dist- packages/ armada/ cli/apply. py", line 255, in handle manifest= self.target_ manifest) lib/python3. 6/dist- packages/ armada/ handlers/ armada. py", line 87, in __init__ values) .update_ manifests( ) lib/python3. 6/dist- packages/ armada/ handlers/ override. py", line 146, in update_manifests yaml_file( value) lib/python3. 6/dist- packages/ armada/ handlers/ override. py", line 40, in _load_yaml_file exceptions. InvalidOverride FileException( doc) exceptions. override_ exceptions. InvalidOverride FileException: /overrides/ platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml is not a valid override file.
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli with open(doc) as f:
2020-06-15 22:05:55.886 56 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli During handling of the above exception, another exception occurred:
2020-06-15 22:05:55.886 56 ERROR armada.cli
2020-06-15 22:05:55.886 56 ERROR armada.cli Traceback (most recent call last):
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli self.invoke()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli resp = self.handle(
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli return future.result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/
2020-06-15 22:05:55.886 56 ERROR armada.cli return self.__get_result()
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/
2020-06-15 22:05:55.886 56 ERROR armada.cli raise self._exception
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/lib/
2020-06-15 22:05:55.886 56 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli target_
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli values=
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli merging_values = self._load_
2020-06-15 22:05:55.886 56 ERROR armada.cli File "/usr/local/
2020-06-15 22:05:55.886 56 ERROR armada.cli raise override_
2020-06-15 22:05:55.886 56 ERROR armada.cli armada.
2020-06-15 22:05:55.886 56 ERROR armada.cli ^[[00m
Test Activity
-------------
Install and Config