StarlingX

  1. Bug #1848721
  2. Comment #0

Comment 0 for bug 1848721

Revision history for this message
Anujeyan Manokeran (anujeyan) wrote : AIO DX+N platform-integ-apps application apply failure

Brief Description
-----------------.
   AIO DX +N system was failing on platform-integ-apps apply . This was an intermittent issue during the install. Reapply was successful.
system application-show platform-integ-apps
+---------------+------------------------------------------+
| Property | Value |
 +---------------+------------------------------------------+
| active | False |
| app_version | 1.0-8 |
| created_at | 2019-10-18T00:48:05.144205+00:00 |
| manifest_file | manifest.yaml |
| manifest_name | platform-integration-manifest |
| name | platform-integ-apps |
| progress | operation aborted, check logs for detail |
| status | apply-failed |
| updated_at | 2019-10-18T01:20:29.208565+00:00 |
 +---------------+------------------------------------------+

2019-10-18 01:18:29.843 46 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176^[[00m
2019-10-18 01:19:29.901 46 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176^[[00m
2019-10-18 01:20:28.108 46 ERROR armada.handlers.wait [-] [chart=kube-system-rbd-provisioner]: Timed out waiting for pods (namespace=kube-system, labels=(app=rbd-provisioner)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`?^[[00m
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada [-] Chart deploy [kube-system-rbd-provisioner] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=kube-system, labels=(app=rbd-provisioner)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`?
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada Traceback (most recent call last):
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada result = get_result()
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 236, in <lambda>
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada if (handle_result(chart, lambda: deploy_chart(chart))):
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases)
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 248, in execute
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada chart_wait.wait(timer)
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, in wait
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada wait.wait(timeout=timeout)
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, in wait
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada modified = self._wait(deadline)
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, in _wait
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada raise k8s_exceptions.KubernetesWatchTimeoutException(error)
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=kube-system, labels=(app=rbd-provisioner)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`?
2019-10-18 01:20:28.109 46 ERROR armada.handlers.armada ^[[00m
2019-10-18 01:20:28.111 46 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['kube-system-rbd-provisioner']^[[00m
2019-10-18 01:20:28.963 46 INFO armada.handlers.lock [-] Releasing lock^[[00m
2019-10-18 01:20:28.968 46 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['kube-system-rbd-provisioner']
2019-10-18 01:20:28.968 46 ERROR armada.cli Traceback (most recent call last):
2019-10-18 01:20:28.968 46 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke

Severity
--------
Major

Steps to Reproduce
------------------
1. As per install procedure install the AIO DX +N system unlock all the nodes.
2. Upload platform-integ-apps
3. Apply platform-integ-apps

System Configuration
--------------------
AIODX+N woflpass-8-12

Expected Behavior
------------------
Apply success no errors without re-apply

Actual Behavior
----------------
As per description apply failure

Reproducibility
---------------
intermittent mostly seen on AIO dx

Branch/Pull Time/Commit
-----------------------
BUILD_DATE= 2019-10-16 20:02:12 -0400

Last Pass
---------
2019-10-10_20-00-00

Timestamp/Logs
--------------
2019-10-18 01:18:29

Test Activity
-------------
Regression test