Helm Charts are not available when running system application-update for kubevirt

[Gleb Aronsky]

Brief Description

When attempting to upgrade KubeVirt using the AppFramework upgrade process (i.e. system application-update), the upgrade fails. The flux helm controller has an error message indicates that the chart is not available. This issue occurs despite both the custom updated build and the installed version of KubeVirt (from the master branch) install without issues when done individually. However, the upgrade process consistently fails.

Severity

Major

Steps to Reproduce

Install KubeVirt Application from Master Branch:
Use a recent ISO build, which should contain the lastest version of KubeVirt from the master branch.
Verify Successful Installation and Stabilization of KubeVirt:
Ensure that the KubeVirt application installs successfully.
Wait for the application to stabilize, which typically takes around 3 minutes.
Initiate Upgrade Using the Designer Build:
With KubeVirt installed and stable, execute the system application-upgrade command using the designer build.
Expected Behavior

The KubeVirt and CDI, both components of the kubevirt-app, should upgrade successfully. Successful upgrade can be verified by ensuring all pods are running the latest versions: 1.1.0 for KubeVirt and 1.57.0 for CDI.

Actual Behavior

Master branch version of kubevirt installed:

[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+--------------------------+-----------+-------------------------------------------+------------------+----------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+-----------+-------------------------------------------+------------------+----------+-----------+
| cert-manager | 23.09-72 | cert-manager-fluxcd-manifests | fluxcd-manifests | applied | completed |
| dell-storage | 23.09-7 | dell-storage-fluxcd-manifests | fluxcd-manifests | uploaded | completed |
| kubevirt-app | 23.09-14 | kubevirt-app-fluxcd-manifests | fluxcd-manifests | applied | completed |
| nginx-ingress-controller | 23.09-52 | nginx-ingress-controller-fluxcd-manifests | fluxcd-manifests | applied | completed |
| oidc-auth-apps | 23.09-47 | oidc-auth-apps-fluxcd-manifests | fluxcd-manifests | uploaded | completed |
| platform-integ-apps | 23.09-126 | platform-integ-apps-fluxcd-manifests | fluxcd-manifests | applied | completed |
+--------------------------+-----------+-------------------------------------------+------------------+----------+-----------+
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get all -n kubevirt
Warning: kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2.
NAME READY STATUS RESTARTS AGE
pod/virt-api-57db986b8f-qhwn5 1/1 Running 0 2m12s
pod/virt-controller-7bccf56b45-9vn9z 1/1 Running 0 107s
pod/virt-handler-cn7qm 2/2 Running 0 107s
pod/virt-operator-7d4655f78d-rnd8k 1/1 Running 0 2m47sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubevirt-operator-webhook ClusterIP 10.98.73.70 <none> 443/TCP 2m15s
service/kubevirt-prometheus-metrics ClusterIP None <none> 443/TCP 2m15s
service/virt-api ClusterIP 10.101.76.196 <none> 443/TCP 2m15s
service/virt-exportproxy ClusterIP 10.102.104.85 <none> 443/TCP 2m15sNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/virt-handler 1 1 1 1 1 kubernetes.io/os=linux 107sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/virt-api 1/1 1 1 2m12s
deployment.apps/virt-controller 1/1 1 1 107s
deployment.apps/virt-operator 1/1 1 1 2m47sNAME DESIRED CURRENT READY AGE
replicaset.apps/virt-api-57db986b8f 1 1 1 2m12s
replicaset.apps/virt-controller-7bccf56b45 1 1 1 107s
replicaset.apps/virt-operator-7d4655f78d 1 1 1 2m47sNAME AGE PHASE
kubevirt.kubevirt.io/kubevirt 2m47s Deployed

[sysadmin@controller-0 ~(keystone_admin)]$ helm list -A
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/admin.conf
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cm-cert-manager cert-manager 1 2024-01-16 17:08:35.69711518 +0000 UTC deployed cert-manager-v1.13.1 v1.13.1
deployment-manager default 1 2024-01-16 17:11:05.670897014 +0000 UTC deployed wind-river-cloud-platform-deployment-manager-2.0.10 1.0
dm-monitor platform-deployment-manager 1 2024-01-16 17:11:11.986691203 +0000 UTC deployed dm-monitor-1.0.0 1.0.0
ic-nginx-ingress kube-system 1 2024-01-16 17:07:19.825750451 +0000 UTC deployed ingress-nginx-4.8.2 1.9.3
kubevirt-app kube-system 1 2024-01-17 08:20:52.985657931 +0000 UTC deployed kubevirt-app-0.1.0 1.0.0
stx-ceph-pools-audit kube-system 1 2024-01-16 17:29:08.782156146 +0000 UTC deployed ceph-pools-audit-0.1.0 1.0
stx-cephfs-provisioner kube-system 1 2024-01-16 17:29:08.783244563 +0000 UTC deployed ceph-csi-cephfs-3.9.0 3.9.0
stx-rbd-provisioner kube-system 1 2024-01-16 17:29:08.868377259 +0000 UTC deployed ceph-csi-rbd-3.9.0 3.9.0
Updating the application I observer errors in the flux-helm controller.

[sysadmin@controller-0 helm(keystone_admin)]$ system application-update ./kubevirt-app-1.0-17.tgz2024-01-17T08:22:00.481Z

[sysadmin@controller-0 helm(keystone_admin)]$ kubectl logs -n flux-helm source-controller-5db7798746-n5f5w -f |grep -i kubevirt |grep -i error
2024-01-17T08:22:00.481Z ERROR reconciliation stalled {"controller": "helmchart", "controllerGroup": "source.toolkit.fluxcd.io", "controllerKind": "HelmChart", "HelmChart": {"name":"kube-system-kubevirt-app","namespace":"kube-system"}, "namespace": "kube-system", "name": "kube-system-kubevirt-app", "reconcileID": "9e20eebe-088d-4206-a347-02f51848ce2a", "error": "invalid chart reference: failed to get chart version for remote reference: no 'kubevirt-app' chart with version matching '1.1.0' found"}

[sysadmin@controller-0 helm(keystone_admin)]$ kubectl logs -n flux-helm helm-controller-6c795c895c-r2c8v -f |grep -i kubevirt |grep -i error
2024-01-17T08:10:27.440Z ERROR Reconciler error {"controller": "helmrelease", "controllerGroup": "helm.toolkit.fluxcd.io", "controllerKind": "HelmRelease", "HelmRelease": {"name":"kubevirt-app","namespace":"kube-system"}, "namespace": "kube-system", "name": "kubevirt-app", "reconcileID": "b29fec30-33b0-46a7-aa45-1d89da838087", "error": "Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"kubevirt-app\": StorageError: invalid object, Code: 4, Key: /registry/helm.toolkit.fluxcd.io/helmreleases/kube-system/kubevirt-app, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: ac9b047b-baf5-45ed-a842-39ba83885629, UID in object meta: "}

Flux trying to revert the version back (master build), but it's failing because the init container of the master build can't handle the update contents of /var/opt/kubevirt. This is a follow-on effect to the upgrade failure.

[sysadmin@controller-0 helm(keystone_admin)]$ kubectl get all -n kubevirt
Warning: kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2.
NAME READY STATUS RESTARTS AGE
pod/virt-api-7c45bff6b6-lwhkw 1/1 Running 0 54s
pod/virt-controller-5b789f8b65-b6ntz 1/1 Running 0 80s
pod/virt-handler-57qbk 2/2 Running 0 105s
pod/virt-operator-6fd659494-wx854 1/1 Running 1 (11s ago) 2m23s
pod/virt-operator-7d4655f78d-k86wq 0/1 Init:Error 2 (22s ago) 23sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubevirt-operator-webhook ClusterIP 10.98.73.70 <none> 443/TCP 13m
service/kubevirt-prometheus-metrics ClusterIP None <none> 443/TCP 13m
service/virt-api ClusterIP 10.101.76.196 <none> 443/TCP 13m
service/virt-exportproxy ClusterIP 10.102.104.85 <none> 443/TCP 13mNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/virt-handler 1 1 1 1 1 kubernetes.io/os=linux 13mNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/virt-api 1/1 1 1 13m
deployment.apps/virt-controller 1/1 1 1 13m
deployment.apps/virt-operator 1/1 1 1 14mNAME DESIRED CURRENT READY AGE
replicaset.apps/virt-api-57db986b8f 0 0 0 13m
replicaset.apps/virt-api-7c45bff6b6 1 1 1 10m
replicaset.apps/virt-controller-5b789f8b65 1 1 1 11m
replicaset.apps/virt-controller-7bccf56b45 0 0 0 13m
replicaset.apps/virt-operator-6fd659494 1 1 1 12m
replicaset.apps/virt-operator-7d4655f78d 1 1 0 14m

Reproducibility

100%