oidc app apply failed on a upgraded system

[Michel Thebeau [WIND]]

Brief Description

oidc app apply failed with an error on upgraded system

Severity

major

Steps to Reproduce

1)Run the job deploy oidc-auth-apps on an upgraded system

2)the app apply fails with the following error in sysinv.log

sysinv 2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app [-] Helm operation failure: Failed to retrieve helmv2 releases: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1: sysinv.common.exception.HelmTillerFailure: Helm operation failure: Failed to retrieve helmv2 releases: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/helm/utils.py", line 132, in retrieve_helm_v2_releases
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app raise exception.HelmTillerFailure(
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app sysinv.common.exception.HelmTillerFailure: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app During handling of the above exception, another exception occurred:
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 1527, in _update_app_releases_version
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app deployed_releases = helm_utils.retrieve_helm_releases()
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/helm/utils.py", line 166, in retrieve_helm_releases
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app deployed_releases.update(retrieve_helm_v2_releases())
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/helm/utils.py", line 150, in retrieve_helm_v2_releases
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app raise exception.HelmTillerFailure(
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app sysinv.common.exception.HelmTillerFailure: Helm operation failure: Failed to retrieve helmv2 releases: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1
2022-12-01 19:31:22.227 942456 ERROR sysinv.conductor.kube_app
sysinv 2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app [-] Failed to update/record application Helm operation failure: Failed to retrieve helmv2 releases: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1 releases' versions.: sysinv.common.exception.SysinvException: Failed to update/record application Helm operation failure: Failed to retrieve helmv2 releases: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1 releases' versions.
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/helm/utils.py", line 132, in retrieve_helm_v2_releases
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app raise exception.HelmTillerFailure(
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app sysinv.common.exception.HelmTillerFailure: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app During handling of the above exception, another exception occurred:
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 1527, in _update_app_releases_version
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app deployed_releases = helm_utils.retrieve_helm_releases()
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/helm/utils.py", line 166, in retrieve_helm_releases
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app deployed_releases.update(retrieve_helm_v2_releases())
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/helm/utils.py", line 150, in retrieve_helm_v2_releases
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app raise exception.HelmTillerFailure(
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app sysinv.common.exception.HelmTillerFailure: Helm operation failure: Failed to retrieve helmv2 releases: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app During handling of the above exception, another exception occurred:
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 3039, in perform_app_apply
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app self._update_app_releases_version(app.name)
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 1544, in _update_app_releases_version
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app raise exception.SysinvException(_(
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app sysinv.common.exception.SysinvException: Failed to update/record application Helm operation failure: Failed to retrieve helmv2 releases: Helm operation failure: helmv2-cli -- helm list operation failed without error message, errno=1 releases' versions.
2022-12-01 19:31:22.229 942456 ERROR sysinv.conductor.kube_app

Expected Behavior

oidc app apply should work fine on upgraded system

Actual Behavior

oidc app apply fails with an error

Reproducibility

Yes

System Configuration

Starlingx master

Last Pass

In this release upgrade test probably this is the first we are testing oidc apply on a upgraded system

Timestamp/Logs

Per description

Alarms

na

Test Activity

upgrade testing

Workaround

Probably: manually remove the old oisc-auth-apps application and upload the new version before trying to use it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/866405
Committed: https://opendev.org/starlingx/config/commit/3ca418084e790f34a483f6d19c62c4c97f505a12
Submitter: "Zuul (22348)"
Branch: master

commit 3ca418084e790f34a483f6d19c62c4c97f505a12
Author: Michel Thebeau <email address hidden>
Date: Thu Dec 1 17:32:31 2022 -0500

    handle oidc-auth-apps during upgrade

    After upgrade the oidc-auth-apps application is still at 22.06 release
    version. oidc-auth-apps is excluded from 65-k8s-app-upgrade.sh for
    upgrade from 21.12 to 22.12. But it should be run for 22.06 to 22.12
    upgrade.

    Closes-bug: 1999093

    Test-plan
    PASS: unit test, manually tested
    PASS: AIO-SX upgrade 22.06 to 22.12, apply oidc-auth-apps

    Change-Id: Ib108951da3f216ca5a2eb1dd6d36a2bd098bc79a
    Signed-off-by: Michel Thebeau <email address hidden>