AIO-DX upgrade: Activation fails with error failed to find plugin "multus"

[Lucas]

Brief Description
------------------

Upgrade from 21.12 Centos to 22.06 Centos fails at activation.

Severity
------------------

Critical

Steps to Reproduce
------------------

Issue was detected using an automated vbox upgrade

Expected Behavior
------------------

Activation process finish successfully.

Actual Behavior
------------------

Activation process fails.

Reproducibility
------------------

Intermittent

Using automation seen most of the time, very high chance of reproduction.

System Configuration
------------------

Last Pass
------------------

Same loads passed other times.

Timestamp/Logs
------------------

/var/log/platform.log

```
TASK [common/fluxcd-controllers : Fail if the helm and source controllers are not ready by this time] ***
Saturday 30 July 2022 03:55:38 +0000 (0:03:59.810) 0:04:18.696 *********
failed: [localhost] (item={'_ansible_parsed': True, 'stderr_lines': [u'error: timed out waiting for the condition on deployments/helm-controller'], u'changed': True, u'stderr': u'error: timed out waiting for the condition on deployments/helm-controller', u'ansible_job_id': u'443083077343.212220', u'stdout': u'', '_ansible_item_result': True, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment helm-controller --timeout=240s', u'removes': None, u'argv': None, u'warn': True, u'chdir': None, u'stdin': None}}, 'attempts': 39, u'delta': u'0:04:01.541827', 'stdout_lines': [], 'failed_when_result': False, '_ansible_no_log': False, u'end': u'2022-07-30 03:55:37.644366', '_ansible_item_label': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_item_label':

{u'namespace': u'flux-helm', u'deployment': u'helm-controller'}
, u'ansible_job_id': u'443083077343.212220', 'item':

{u'namespace': u'flux-helm', u'deployment': u'helm-controller'}
, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/443083077343.212220', '_ansible_ignore_errors': None, '_ansible_no_log': False}, u'start': u'2022-07-30 03:51:36.102539', u'cmd': [u'kubectl', u'--kubeconfig=/etc/kubernetes/admin.conf', u'wait', u'--namespace=flux-helm', u'--for=condition=Available', u'deployment', u'helm-controller', u'--timeout=240s'], u'finished': 1, u'failed': False, 'item': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'443083077343.212220', 'item':

{u'namespace': u'flux-helm', u'deployment': u'helm-controller'}
, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/443083077343.212220', '_ansible_ignore_errors': None, '_ansible_item_label': {u'namespace': u'flux-helm', u'deployment': u'helm-controller'}}, u'rc': 1, u'msg': u'non-zero return code', '_ansible_ignore_errors': None}) => changed=false
item:
ansible_job_id: '443083077343.212220'
attempts: 39
changed: true
cmd:

kubectl
--kubeconfig=/etc/kubernetes/admin.conf
wait
--namespace=flux-helm
--for=condition=Available
deployment
helm-controller
--timeout=240s
delta: '0:04:01.541827'
end: '2022-07-30 03:55:37.644366'
failed: false
failed_when_result: false
finished: 1
invocation:
module_args:
_raw_params: kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment helm-controller --timeout=240s
_uses_shell: false
argv: null
chdir: null
creates: null
executable: null
removes: null
stdin: null
warn: true
item:
ansible_job_id: '443083077343.212220'
changed: true
failed: false
finished: 0
item:
deployment: helm-controller
namespace: flux-helm
results_file: /root/.ansible_async/443083077343.212220
started: 1
msg: non-zero return code
rc: 1
start: '2022-07-30 03:51:36.102539'
stderr: 'error: timed out waiting for the condition on deployments/helm-controller'
stderr_lines:
'error: timed out waiting for the condition on deployments/helm-controller'
stdout: ''
stdout_lines: []
msg: 'Pod {u''namespace'': u''flux-helm'', u''deployment'': u''helm-controller''}
is still not ready.'
failed: [localhost] (item={'_ansible_parsed': True, 'stderr_lines': [u'error: timed out waiting for the condition on deployments/source-controller'], u'changed': True, u'stderr': u'error: timed out waiting for the condition on deployments/source-controller', u'ansible_job_id': u'471441098672.212247', u'stdout': u'', '_ansible_item_result': True, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment source-controller --timeout=240s', u'removes': None, u'argv': None, u'warn': True, u'chdir': None, u'stdin': None}}, 'attempts': 1, u'delta': u'0:04:00.555638', 'stdout_lines': [], 'failed_when_result': False, '_ansible_no_log': False, u'end': u'2022-07-30 03:55:38.016507', '_ansible_item_label':

Unknown macro: {'_ansible_parsed'}
, u'start': u'2022-07-30 03:51:37.460869', u'cmd': [u'kubectl', u'--kubeconfig=/etc/kubernetes/admin.conf', u'wait', u'--namespace=flux-helm', u'--for=condition=Available', u'deployment', u'source-controller', u'--timeout=240s'], u'finished': 1, u'failed': False, 'item': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'471441098672.212247', 'item':

{u'namespace': u'flux-helm', u'deployment': u'source-controller'}
, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/471441098672.212247', '_ansible_ignore_errors': None, '_ansible_item_label': {u'namespace': u'flux-helm', u'deployment': u'source-controller'}}, u'rc': 1, u'msg': u'non-zero return code', '_ansible_ignore_errors': None}) => changed=false
item:
ansible_job_id: '471441098672.212247'
attempts: 1
changed: true
cmd:

kubectl
--kubeconfig=/etc/kubernetes/admin.conf
wait
--namespace=flux-helm
--for=condition=Available
deployment
source-controller
--timeout=240s
delta: '0:04:00.555638'
end: '2022-07-30 03:55:38.016507'
failed: false
failed_when_result: false
finished: 1
invocation:
module_args:
_raw_params: kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment source-controller --timeout=240s
_uses_shell: false
argv: null
chdir: null
creates: null
executable: null
removes: null
stdin: null
warn: true
item:
ansible_job_id: '471441098672.212247'
changed: true
failed: false
finished: 0
item:
deployment: source-controller
namespace: flux-helm
results_file: /root/.ansible_async/471441098672.212247
started: 1
msg: non-zero return code
rc: 1
start: '2022-07-30 03:51:37.460869'
stderr: 'error: timed out waiting for the condition on deployments/source-controller'
stderr_lines:
'error: timed out waiting for the condition on deployments/source-controller'
stdout: ''
stdout_lines: []
msg: 'Pod {u''namespace'': u''flux-helm'', u''deployment'': u''source-controller''}
is still not ready.'
```

Alarms
------------------

Test Activity
------------------

Workaround
------------------

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/855364
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/7e5861e42210948580b3c0556007d5bb769c58a3
Submitter: "Zuul (22348)"
Branch: master

commit 7e5861e42210948580b3c0556007d5bb769c58a3
Author: Lucas Cavalcante <email address hidden>
Date: Wed Aug 31 09:39:27 2022 -0300

    Fix upgrade fails with k8s-networking error

    AIO-DX upgrade: Activation fails with error failed to find plugin "multus"

    Upgrade from 21.12 Centos to 22.06 Centos fails at activation.

    TEST PLAN:
    PASS: Vbox automated upgrade works

    Closes-bug: 1988305
    Signed-off-by: Lucas Cavalcante <email address hidden>
    Change-Id: I9c5c0995cb34fed368cb0bc47608f47853d5a2de