kubernetes upgrade aborts w/ app with timing:post

[Igor Pires Soares]

*+Brief Description+*
When attempting a kubernetes version upgrade(in this scenario 1.27.5 to 1.28.4), the upgrade is aborted due to apps not being compatible with the new version during pre-upgrade-apps step.
*+Severity+*

Minor: System/Feature is usable with minor issue

*+Steps to Reproduce+*
 * copied the metrics-server app .tgz from /usr/local/share/applications helm and updated the metadata to timing:post:

{code:java}
[sysadmin@controller-0 ~(keystone_admin)]$ cat test/pre/metadata.yaml
app_name: metrics-server
app_version: 24.03-54
helm_repo: stx-platform
maintain_user_overrides: true
upgrades:
  auto_update: true
k8s_upgrades:
  auto_update: true
  timing: post
supported_k8s_version:
  minimum: 1.24.4
  maximum: 1.27.5
behavior:
  platform_managed_app: yes {code}
 * created another copy and updated it's metadata to:

{code:java}
app_name: metrics-server
app_version: 24.03-55
helm_repo: stx-platform
maintain_user_overrides: true
upgrades:
  auto_update: true
supported_k8s_version:
  minimum: 1.28.4
  maximum: 1.29.2
behavior:
  platform_managed_app: yes{code}
 * copied back to apps folder, updated ostree/lock, and checked if database is updated:

{code:java}
 2024-05-10 12:11:31.740806 |            |            | 26 | metrics-server                  | 24.03-55  | /usr/local/share/applications/helm/metrics-server-24.03-55.tgz                  | t           | t               | post       | 1.28.4              | 1.29.2              |
 2024-05-10 15:45:08.90804  |            |            | 27 | metrics-server                  | 24.03-54  | /usr/local/share/applications/helm/metrics-server-24.03-54.tgz                  | t           | t               | post       | 1.24.4              | 1.27.5              |
(26 rows) {code}
 * started k8s upgrade orchestration.

*+Expected Behavior+*

k8s version upgrade should complete normally since the app available to be post-updated is compatible with the target version.

*+Actual Behavior+*

k8s version upgrade is aborted

*+Reproducibility+*

Reproducible

*+System Configuration+*

SX and DX

*+Load info (eg: 2022-03-10_20-00-07)+*

SW_VERSION="24.03"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="2024-05-07_19-00-25"
SRC_BUILD_ID="1965"

BUILD_BY="jenkins"
BUILD_NUMBER="740"
BUILD_DATE="2024-05-07 23:00:25 +0000"

*+Last Pass+*

N/A

*+Timestamp/Logs+*

sysinv:
{code:java}
sysinv 2024-05-10 15:56:19.341 13942 ERROR sysinv.conductor.manager [-] The installed Application metrics-server (24.03-54) is incompatible with the new Kubernetes version v1.28.4.
sysinv 2024-05-10 15:56:19.349 13942 INFO sysinv.conductor.manager [-] Failed to update applications to match Kubernetes version v1.28.4. {code}
[^sysinv.log]

k8s upgrade log message:
{code:java}
sysinv 2024-05-10 15:56:19.341 13942 ERROR sysinv.conductor.manager [-] The installed Application metrics-server (24.03-54) is incompatible with the new Kubernetes version v1.28.4.
sysinv 2024-05-10 15:56:19.349 13942 INFO sysinv.conductor.manager [-] Failed to update applications to match Kubernetes version v1.28.4. {code}

*+Alarms+*

upgrade aborted

*+Test Activity+*

Sanity, Feature Testing

*+Workaround+*

N/A

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/919900
Committed: https://opendev.org/starlingx/config/commit/7113c1c99b2632f413e5df938c074cae90d50644
Submitter: "Zuul (22348)"
Branch: master

commit 7113c1c99b2632f413e5df938c074cae90d50644
Author: Igor Soares <email address hidden>
Date: Thu May 16 18:20:23 2024 -0300

    Prevent app incompatibility gap during k8s upgrade

    Introduce an additional check to block Kubernetes upgrades on the
    kube-upgrade-start step if applied apps marked to be post updated are
    incompatible with the target k8s version. This will prevent
    incompatibility gaps in the upgrade process, i.e. avoiding that the
    post-update stage is reached while running an incompatible app.

    Application developers can leverage the auto update feature and the
    pre-update step in case incompatible apps need to be updated before
    deploying a new Kubernetes version.

    Test Plan:
    PASS: build-pkgs -a && build-image.
    PASS: Deploy platform with Kubernetes 1.24.
          Apply a metrics-server version containing
          supported_k8s_version:minimum: 1.24 and
          supported_k8s_version:maximum: 1.27.
          Add a new metrics-server bundle containing
          supported_k8s_version:minimum: 1.28 and
          supported_k8s_version:maximum: 1.29.
          Start Kubernetes upgrade to 1.28.
          Confirm that the upgrade was blocked.
    PASS: Deploy platform with Kubernetes 1.24.
          Apply a metrics-server version containing
          supported_k8s_version:minimum: 1.24 and
          supported_k8s_version:maximum: 1.28.
          Add a new metrics-server bundle containing
          supported_k8s_version:minimum: 1.28 and
          supported_k8s_version:maximum: 1.29.
          Start Kubernetes upgrade to 1.28.
          Confirm that metrics-server is updated during post-update step.

    Closes-Bug: 2066042

    Change-Id: Ie5d7e872ae49f833bf1cf1a4eb80a7f50dad8920
    Signed-off-by: Igor Soares <email address hidden>