AIO-DX upgrade: Activation fails with error failed to find plugin "multus"

Bug #1988305 reported by Lucas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Lucas

Bug Description

Brief Description
------------------

Upgrade from 21.12 Centos to 22.06 Centos fails at activation.

Severity
------------------

Critical

Steps to Reproduce
------------------

Issue was detected using an automated vbox upgrade

Expected Behavior
------------------

Activation process finish successfully.

Actual Behavior
------------------

Activation process fails.

Reproducibility
------------------

Intermittent

Using automation seen most of the time, very high chance of reproduction.

System Configuration
------------------

Last Pass
------------------

Same loads passed other times.

Timestamp/Logs
------------------

/var/log/platform.log

```
TASK [common/fluxcd-controllers : Fail if the helm and source controllers are not ready by this time] ***
Saturday 30 July 2022 03:55:38 +0000 (0:03:59.810) 0:04:18.696 *********
failed: [localhost] (item={'_ansible_parsed': True, 'stderr_lines': [u'error: timed out waiting for the condition on deployments/helm-controller'], u'changed': True, u'stderr': u'error: timed out waiting for the condition on deployments/helm-controller', u'ansible_job_id': u'443083077343.212220', u'stdout': u'', '_ansible_item_result': True, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment helm-controller --timeout=240s', u'removes': None, u'argv': None, u'warn': True, u'chdir': None, u'stdin': None}}, 'attempts': 39, u'delta': u'0:04:01.541827', 'stdout_lines': [], 'failed_when_result': False, '_ansible_no_log': False, u'end': u'2022-07-30 03:55:37.644366', '_ansible_item_label': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_item_label':

{u'namespace': u'flux-helm', u'deployment': u'helm-controller'}
, u'ansible_job_id': u'443083077343.212220', 'item':

{u'namespace': u'flux-helm', u'deployment': u'helm-controller'}
, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/443083077343.212220', '_ansible_ignore_errors': None, '_ansible_no_log': False}, u'start': u'2022-07-30 03:51:36.102539', u'cmd': [u'kubectl', u'--kubeconfig=/etc/kubernetes/admin.conf', u'wait', u'--namespace=flux-helm', u'--for=condition=Available', u'deployment', u'helm-controller', u'--timeout=240s'], u'finished': 1, u'failed': False, 'item': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'443083077343.212220', 'item':

{u'namespace': u'flux-helm', u'deployment': u'helm-controller'}
, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/443083077343.212220', '_ansible_ignore_errors': None, '_ansible_item_label': {u'namespace': u'flux-helm', u'deployment': u'helm-controller'}}, u'rc': 1, u'msg': u'non-zero return code', '_ansible_ignore_errors': None}) => changed=false
item:
ansible_job_id: '443083077343.212220'
attempts: 39
changed: true
cmd:

kubectl
--kubeconfig=/etc/kubernetes/admin.conf
wait
--namespace=flux-helm
--for=condition=Available
deployment
helm-controller
--timeout=240s
delta: '0:04:01.541827'
end: '2022-07-30 03:55:37.644366'
failed: false
failed_when_result: false
finished: 1
invocation:
module_args:
_raw_params: kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment helm-controller --timeout=240s
_uses_shell: false
argv: null
chdir: null
creates: null
executable: null
removes: null
stdin: null
warn: true
item:
ansible_job_id: '443083077343.212220'
changed: true
failed: false
finished: 0
item:
deployment: helm-controller
namespace: flux-helm
results_file: /root/.ansible_async/443083077343.212220
started: 1
msg: non-zero return code
rc: 1
start: '2022-07-30 03:51:36.102539'
stderr: 'error: timed out waiting for the condition on deployments/helm-controller'
stderr_lines:
'error: timed out waiting for the condition on deployments/helm-controller'
stdout: ''
stdout_lines: []
msg: 'Pod {u''namespace'': u''flux-helm'', u''deployment'': u''helm-controller''}
is still not ready.'
failed: [localhost] (item={'_ansible_parsed': True, 'stderr_lines': [u'error: timed out waiting for the condition on deployments/source-controller'], u'changed': True, u'stderr': u'error: timed out waiting for the condition on deployments/source-controller', u'ansible_job_id': u'471441098672.212247', u'stdout': u'', '_ansible_item_result': True, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment source-controller --timeout=240s', u'removes': None, u'argv': None, u'warn': True, u'chdir': None, u'stdin': None}}, 'attempts': 1, u'delta': u'0:04:00.555638', 'stdout_lines': [], 'failed_when_result': False, '_ansible_no_log': False, u'end': u'2022-07-30 03:55:38.016507', '_ansible_item_label':

Unknown macro: {'_ansible_parsed'}
, u'start': u'2022-07-30 03:51:37.460869', u'cmd': [u'kubectl', u'--kubeconfig=/etc/kubernetes/admin.conf', u'wait', u'--namespace=flux-helm', u'--for=condition=Available', u'deployment', u'source-controller', u'--timeout=240s'], u'finished': 1, u'failed': False, 'item': {'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'471441098672.212247', 'item':

{u'namespace': u'flux-helm', u'deployment': u'source-controller'}
, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/471441098672.212247', '_ansible_ignore_errors': None, '_ansible_item_label': {u'namespace': u'flux-helm', u'deployment': u'source-controller'}}, u'rc': 1, u'msg': u'non-zero return code', '_ansible_ignore_errors': None}) => changed=false
item:
ansible_job_id: '471441098672.212247'
attempts: 1
changed: true
cmd:

kubectl
--kubeconfig=/etc/kubernetes/admin.conf
wait
--namespace=flux-helm
--for=condition=Available
deployment
source-controller
--timeout=240s
delta: '0:04:00.555638'
end: '2022-07-30 03:55:38.016507'
failed: false
failed_when_result: false
finished: 1
invocation:
module_args:
_raw_params: kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --namespace=flux-helm --for=condition=Available deployment source-controller --timeout=240s
_uses_shell: false
argv: null
chdir: null
creates: null
executable: null
removes: null
stdin: null
warn: true
item:
ansible_job_id: '471441098672.212247'
changed: true
failed: false
finished: 0
item:
deployment: source-controller
namespace: flux-helm
results_file: /root/.ansible_async/471441098672.212247
started: 1
msg: non-zero return code
rc: 1
start: '2022-07-30 03:51:37.460869'
stderr: 'error: timed out waiting for the condition on deployments/source-controller'
stderr_lines:
'error: timed out waiting for the condition on deployments/source-controller'
stdout: ''
stdout_lines: []
msg: 'Pod {u''namespace'': u''flux-helm'', u''deployment'': u''source-controller''}
is still not ready.'
```

Alarms
------------------

Test Activity
------------------

Workaround
------------------

Lucas (lcavalca)
Changed in starlingx:
assignee: nobody → Lucas (lcavalca)
Changed in starlingx:
status: New → In Progress
Ghada Khalil (gkhalil)
tags: added: stx.update
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/855364
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/7e5861e42210948580b3c0556007d5bb769c58a3
Submitter: "Zuul (22348)"
Branch: master

commit 7e5861e42210948580b3c0556007d5bb769c58a3
Author: Lucas Cavalcante <email address hidden>
Date: Wed Aug 31 09:39:27 2022 -0300

    Fix upgrade fails with k8s-networking error

    AIO-DX upgrade: Activation fails with error failed to find plugin "multus"

    Upgrade from 21.12 Centos to 22.06 Centos fails at activation.

    TEST PLAN:
    PASS: Vbox automated upgrade works

    Closes-bug: 1988305
    Signed-off-by: Lucas Cavalcante <email address hidden>
    Change-Id: I9c5c0995cb34fed368cb0bc47608f47853d5a2de

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
tags: added: stx.8.0
Changed in starlingx:
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.