AIO-SX upgrade-activate failed due to deployment plugin installation failure

[Susendra Selvaraj]

Brief Description

AIO-SX upgrade from 22.12 prestaging load (2024-02-22_06-00-08) to 24.03 designer load.
The host is upgraded successfully and running 24.03, all services are enabled, no alarms.
The upgrade activation failed with errors:

Tuesday 27 February 2024 10:29:21 +0000 (0:00:00.096) 0:00:14.138 ******
  cmd: KUBECONFIG=/etc/kubernetes/admin.conf /usr/sbin/helm upgrade --install deployment-manager --values /usr/local/share/applications/helm/wind-river-cloud-platform-deployment-manager-2.0.10.tgz /usr/local/share/applications/helm/deployment-manager-24.03-5.tgz
  delta: '0:00:00.114516'
  end: '2024-02-27 10:29:22.434627'
  start: '2024-02-27 10:29:22.320111'
  stderr: |-
    WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/admin.conf
    Error: failed to parse /usr/local/share/applications/helm/wind-river-cloud-platform-deployment-manager-2.0.10.tgz: error converting YAML to JSON: yaml: control characters are not allowed
PLAY RECAP *********************************************************************
localhost : ok=24 changed=10 unreachable=0 failed=1 skipped=22 rescued=0 ignored=3
Tuesday 27 February 2024 10:29:22 +0000 (0:00:00.595) 0:00:14.733 ******
sysadmin@controller-0:~$ ls /usr/local/share/applications/playbooks/*deployment-manager.yaml /usr/local/share/applications/playbooks/wind-river-cloud-platform-deployment-manager.yaml sysadmin@controller-0:~$ ls /usr/local/share/applications/helm/*deployment-manager*.tgz /usr/local/share/applications/helm/deployment-manager-24.03-5.tgz /usr/local/share/applications/helm/wind-river-cloud-platform-deployment-manager-2.0.10.tgz sysadmin@controller-0:~$ ls /usr/local/share/applications/overrides/*deployment-manager-overrides.yaml /usr/local/share/applications/overrides/wind-river-cloud-platform-deployment-manager-overrides.yaml
Severity

Major

Steps to Reproduce

Upgrade an AIO-SX from 22.12 prestaging load to 24.09 master load

Expected Behavior

Upgrade activation to complete

Actual Behavior

Upgrade activation failed

Reproducibility

100%

System Configuration

vbox

Load info (eg: 2022-03-10_20-00-07)

22.12 BUILD_ID="2024-02-22_06-00-08"
24.03 designer load

Last Pass

N/A

Timestamp/Logs

/folk/cgts_logs/CGTS-59549/ALL_NODES_20240228.121915.tar

Alarms

No

Test Activity

Feature Testing

Workaround

Describe workaround if available

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/910705
Committed: https://opendev.org/starlingx/config/commit/a6113eb85c19e5ea51323f78c5936ae7abf5af64
Submitter: "Zuul (22348)"
Branch: master

commit a6113eb85c19e5ea51323f78c5936ae7abf5af64
Author: sharang bhardwaj <email address hidden>
Date: Fri Mar 1 02:06:34 2024 -0500

    Retrieve only the deploy plug-in helm-chart

    Fetching the deploy plug-in helm-chart by using
    the command "grep "deployment-manager.tgz" in the
    directory /usr/local/share/applications/helm/ for
    the purpose of deploying an upgrade to deploy plug-in.

    Due to recent developments in the deploy plug-in
    system-application, a new helm-chart is generated with
    the naming convention deployment-manager_*tgz. The
    current retrieval process is obtaining both
    deploy plug-in and system application helm-charts.

    To address this, a hyphen ("-") has been added to the
    grep command ("-deployment-manager") to specifically
    check for deploy plug-in and exclude the system-application.

    Test-Plan:
    PASS: Verify the deploy plug-in helm-chart is retrieved.
    PASS: Verify the deploy plug-in is refreshed when it
          is installed

    Closes-Bug: 2055699

    Change-Id: I9d8ae5d491f6a9cacca1e68574c4cb4431264342
    Signed-off-by: sharang bhardwaj <email address hidden>