stx-openstack: Cinder helm release fails to upgrade after helm override
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Luan Nunes Utimura |
Bug Description
Brief Description
-----------------
After performing a helm override of the `backup_driver` and `backup_
Severity
--------
Major. After the upgrade fails, the application is left in the `apply-failed` state.
Although the helm release is able to recover after some time (by rolling back), the application will stay in the `apply-failed` state until a second apply is triggered. This can impact some automated tests depending on the timing of those events.
Steps to Reproduce
------------------
1) Upload/apply stx-openstack;
2) Perform a helm override of `backup_driver` and `backup_
```
system helm-override-
```
and re-apply stx-openstack;
3) Delete helm overrides:
```
system helm-override-
```
and re-apply stx-openstack.
Expected Behavior
------------------
The application is successfully applied both times.
Actual Behavior
----------------
The application fails to apply the second time.
Reproducibility
---------------
Reproducible.
System Configuration
-------
AIO-SX.
Branch/Pull Time/Commit
-------
master:
* /mirror/
Last Pass
---------
N/A.
Timestamp/Logs
--------------
sysinv 2023-02-25 08:38:03.607 2806263 ERROR sysinv.
Test Activity
-------------
Developer Testing.
Workaround
----------
Before applying stx-openstack (and after performing the helm overrides), remove the conflicting job manually: `kubectl -n openstack delete job/cinder-
Changed in starlingx: | |
assignee: | nobody → Luan Nunes Utimura (lutimura) |
tags: | added: stx.9.0 stx.distro.openstack |
Changed in starlingx: | |
status: | New → In Progress |
Changed in starlingx: | |
importance: | Undecided → Medium |
Reviewed: https:/ /review. opendev. org/c/starlingx /openstack- armada- app/+/882610 /opendev. org/starlingx/ openstack- armada- app/commit/ 1fb92fb8ba2cc20 1ddc21d90ae12a8 1b1ebf1d42
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 1fb92fb8ba2cc20 1ddc21d90ae12a8 1b1ebf1d42
Author: Luan Nunes Utimura <email address hidden>
Date: Mon May 8 14:53:40 2023 -0300
Fixing cinder helm release storage bootstrap hooks
After performing a helm override of the `backup_driver` and mount_options` fields in `.Values. conf.cinder. DEFAULT` , it has cinder` fails to upgrade due to backup- storage- init` job spec.
`backup_
been observed that `osh-openstack-
changes to immutable fields in `cinder-
Since this is only a problem because the job is being kept in the system
even after it has finished its task, one can avoid this upgrade failure
by simply leveraging helm hooks [1].
[1] https:/ /helm.sh/ docs/topics/ charts_ hooks/
Test Plan: helm-fluxcd package
`kubectl -n openstack get jobs -w | grep cinder`
PASS - Build openstack-helm package
PASS - Build stx-openstack-
PASS - Build stx-openstack helm charts
PASS - Upload/apply stx-openstack
PASS - While watching the jobs:
Perform a helm override and verify that storage bootstrap jobs
are being terminated and reinitialized during the helm release
upgrade
PASS - Remove/delete stx-openstack
Closes-Bug: 2018930
Change-Id: Icfda326ac39056 4a3bca1c358e8e4 44b95d66808
Signed-off-by: Luan Nunes Utimura <email address hidden>