Investigations showed that tiller container started executing commands before NDP finished.
Severity
--------
Critical: System/Feature is not usable due to the defect
Steps to Reproduce
------------------
Run an AIO-SX upgrade.
In fact scale the armada-pod in a loop
kubectl --kubeconfig=/etc/kubernetes/admin.conf scale deployment -n armada armada-api --replicas=0
kubectl --kubeconfig=/etc/kubernetes/admin.conf scale deployment -n armada armada-api --replicas=1
Expected Behavior
------------------
Write down what was expected after taking the steps written above
Actual Behavior
----------------
State what is the actual behavior
Reproducibility
---------------
Once, but tiller problem can be reproduced when running a script in a loop fir several hours.
System Configuration
--------------------
AIO-SX IPv6
Branch/Pull Time/Commit
-----------------------
Any April 2021 load or older
Last Pass
---------
Not relevant
Timestamp/Logs
--------------
2021-04-27 20:30:13,563 p=21107 u=sysadmin | TASK [bootstrap/bringup-essential-services : Fail if any of the Kubernetes component, Networking or Armada pods are not ready by this time] ***************************************************************************
2021-04-27 20:30:13,698 p=21107 u=sysadmin | failed: [localhost] (item={'_ansible_parsed': True, 'stderr_lines': [u'error: timed out waiting for the condition on deployments/armada-api'], u'changed': True, u'stderr': u'error: timed out waiting for the condition on deployments/armada-api', u'ansible_job_id': u'168691713518.173381', u'stdout': u'', '_ansible_item_result': True, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=armada --for=condition=Available deployment armada-api --timeout=30s', u'removes': None, u'argv': None, u'warn': True, u'chdir': None, u'stdin': None}}, 'attempts': 6, u'delta': u'0:00:30.082435', 'stdout_lines': [], 'failed_when_result': False, '_ansible_no_log': False, u'end': u'2021-04-27 20:30:09.129848', '_ansible_item_label': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_item_label':
{u'namespace': u'armada', u'deployment': u'armada-api'}
, u'ansible_job_id': u'168691713518.173381', 'item': {u'namespace': u'armada', u'deployment': u'armada-api'}, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/168691713518.173381', '_ansible_ignore_errors': None, '_ansible_no_log': False}, u'start': u'2021-04-27 20:29:39.047413', u'cmd': [u'kubectl', u'--kubeconfig=/etc/kubernetes/admin.conf', u'wait', u'--namespace=armada', u'--for=condition=Available', u'deployment', u'armada-api', u'--timeout=30s'], u'finished': 1, u'failed': False, 'item': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'168691713518.173381', 'item':
{u'namespace': u'armada', u'deployment': u'armada-api'}
, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/168691713518.173381', '_ansible_ignore_errors': None, '_ansible_item_label': {u'namespace': u'armada', u'deployment': u'armada-api'}}, u'rc': 1, u'msg': u'non-zero return code', '_ansible_ignore_errors': None}) => {"changed": false, "item": {"ansible_job_id": "168691713518.173381", "attempts": 6, "changed": true, "cmd": ["kubectl", "--kubeconfig=/etc/kubernetes/admin.conf", "wait", "--namespace=armada", "--for=condition=Available", "deployment", "armada-api", "--timeout=30s"], "delta": "0:00:30.082435", "end": "2021-04-27 20:30:09.129848", "failed": false, "failed_when_result": false, "finished": 1, "invocation": {"module_args": {"_raw_params": "kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=armada --for=condition=Available deployment armada-api --timeout=30s", "_uses_shell": false, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "warn": true}}, "item": {"ansible_job_id": "168691713518.173381", "changed": true, "failed": false, "finished": 0, "item":
{"deployment": "armada-api", "namespace": "armada"}
, "results_file": "/root/.ansible_async/168691713518.173381", "started": 1}, "msg": "non-zero return code", "rc": 1, "start": "2021-04-27 20:29:39.047413", "stderr": "error: timed out waiting for the condition on deployments/armada-api", "stderr_lines": ["error: timed out waiting for the condition on deployments/armada-api"], "stdout": "", "stdout_lines": []}, "msg": "Pod {u'namespace': u'armada', u'deployment': u'armada-api'} is still not ready."}
2021-04-27 20:30:13,715 p=21107 u=sysadmin | PLAY RECAP ************************************************************************************************************************************************************************************************************
2021-04-27 20:30:13,715 p=21107 u=sysadmin | localhost : ok=432 changed=239 unreachable=0 failed=1
Test Activity
-------------
Testing upgrades
Workaround
-------------
Reinstall the iso and retry upgrade_platform.yaml
Fix proposed to branch: master /review. opendev. org/c/starlingx /ansible- playbooks/ +/790863
Review: https:/