stx-openstack: `cinder-volume-usage-audit` pod stuck on Init:0/2

Bug #2032703 reported by Luan Nunes Utimura
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Low
Luan Nunes Utimura

Bug Description

Brief Description
-----------------
It has been observed that, after an apply of stx-openstack on a system with HTTPS enabled, the `cinder-volume-usage-audit` pod is getting stuck in the "Init:0/2" state.

Severity
--------
Minor.

Steps to Reproduce
------------------
On a system with HTTPS enabled:
1) Upload/apply stx-openstack;
2) Verify that the `cinder-volume-usage-audit` pod was unable to initialize.

Expected Behavior
------------------
After stx-openstack is applied, all pods should be either `running` or `completed`.

Actual Behavior
----------------
The `cinder-volume-usage-audit` pod is stuck in "Init:0/2".

Reproducibility
---------------
Reproducible.

System Configuration
--------------------
AIO-DX with HTTPS enabled.

Branch/Pull Time/Commit
-----------------------
StarlingX (master)
StarlingX OpenStack (master)

Last Pass
---------
N/A.

Timestamp/Logs
--------------
```
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 11h default-scheduler Successfully assigned openstack/cinder-volume-usage-audit-28211225-4g895 to controller-1
  Warning FailedMount 11h (x4 over 11h) kubelet MountVolume.SetUp failed for volume "cinder-tls-api" : failed to sync secret cache: timed out waiting for the condition
  Warning FailedMount 4h56m (x32 over 11h) kubelet Unable to attach or mount volumes: unmounted volumes=[cinder-tls-api], unattached volumes=[pod-tmp etccinder cinder-etc cinder-bin cinder-tls-api kube-api-access-bqcq5]: timed out waiting for the condition
  Warning FailedMount 125m (x48 over 10h) kubelet Unable to attach or mount volumes: unmounted volumes=[cinder-tls-api], unattached volumes=[kube-api-access-bqcq5 pod-tmp etccinder cinder-etc cinder-bin cinder-tls-api]: timed out waiting for the condition
  Warning FailedMount 55m (x35 over 11h) kubelet Unable to attach or mount volumes: unmounted volumes=[cinder-tls-api], unattached volumes=[cinder-bin cinder-tls-api kube-api-access-bqcq5 pod-tmp etccinder cinder-etc]: timed out waiting for the condition
  Warning FailedMount 35m (x47 over 10h) kubelet Unable to attach or mount volumes: unmounted volumes=[cinder-tls-api], unattached volumes=[etccinder cinder-etc cinder-bin cinder-tls-api kube-api-access-bqcq5 pod-tmp]: timed out waiting for the condition
  Warning FailedMount 15m (x51 over 11h) kubelet Unable to attach or mount volumes: unmounted volumes=[cinder-tls-api], unattached volumes=[cinder-tls-api kube-api-access-bqcq5 pod-tmp etccinder cinder-etc cinder-bin]: timed out waiting for the condition
  Warning FailedMount 6m3s (x55 over 11h) kubelet Unable to attach or mount volumes: unmounted volumes=[cinder-tls-api], unattached volumes=[cinder-etc cinder-bin cinder-tls-api kube-api-access-bqcq5 pod-tmp etccinder]: timed out waiting for the condition
  Warning FailedMount 27s (x334 over 11h) kubelet MountVolume.SetUp failed for volume "cinder-tls-api" : secret "cinder-tls-api" not found
```

Test Activity
-------------
Sanity.

Workaround
----------
N/A.

tags: added: stx.9.0 stx.distro.openstack
Changed in starlingx:
status: New → In Progress
Changed in starlingx:
assignee: nobody → Luan Nunes Utimura (lutimura)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-armada-app (master)

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/892418
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/be56c15bc043528f4448ae607182d97641fc8766
Submitter: "Zuul (22348)"
Branch: master

commit be56c15bc043528f4448ae607182d97641fc8766
Author: Luan Nunes Utimura <email address hidden>
Date: Tue Aug 22 20:59:20 2023 -0300

    Fix TLS vol. in cinder-volume-usage-audit CronJob

    After the recent upversion of openstack-helm [1], it has been observed
    that the `cinder-volume-usage-audit` pod is having problems booting on
    systems with HTTPS enabled due to a misconfigured TLS-related
    volume/volumeMount pair.

    Apparently, this pair of volume and volumeMount was introduced with the
    upversion of openstack-helm, and ended up being left out of the changes
    made by patch `0010-Remove-TLS-from-openstack-services.patch` that, in
    theory, would have solved the problem.

    Therefore, this change aims to update the patch in question -- along
    with any other patches to avoid conflicts -- so that the
    `cinder-volume-usage-audit` pod no longer has problems booting on
    systems with HTTPS enabled.

    [1] https://opendev.org/starlingx/openstack-armada-app/commit/8254cd31bb1f12eebc48b712b33f75b2fc0aa571

    Test Plan (on AIO-DX with HTTPS enabled):
    PASS - Build openstack-helm package
    PASS - Build stx-openstack-helm-fluxcd package
    PASS - Build stx-openstack helm charts
    PASS - Upload/apply stx-openstack
    PASS - Verify that all pods -- including `cinder-volume-usage-audit` --
           are either "Running" or "Completed"
    PASS - Remove/delete stx-openstack

    Closes-Bug: 2032703

    Change-Id: Ic13c6945cc9e43f9153820297e74623520446fcd
    Signed-off-by: Luan Nunes Utimura <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-armada-app (f/antelope)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-armada-app (f/antelope)

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/892587
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/0c5e6b143965a6b58dcad6de23d67f1d70c58a20
Submitter: "Zuul (22348)"
Branch: f/antelope

commit 0c5e6b143965a6b58dcad6de23d67f1d70c58a20
Author: Luan Nunes Utimura <email address hidden>
Date: Tue Aug 22 20:59:20 2023 -0300

    Fix TLS vol. in cinder-volume-usage-audit CronJob

    After the recent upversion of openstack-helm [1], it has been observed
    that the `cinder-volume-usage-audit` pod is having problems booting on
    systems with HTTPS enabled due to a misconfigured TLS-related
    volume/volumeMount pair.

    Apparently, this pair of volume and volumeMount was introduced with the
    upversion of openstack-helm, and ended up being left out of the changes
    made by patch `0010-Remove-TLS-from-openstack-services.patch` that, in
    theory, would have solved the problem.

    Therefore, this change aims to update the patch in question -- along
    with any other patches to avoid conflicts -- so that the
    `cinder-volume-usage-audit` pod no longer has problems booting on
    systems with HTTPS enabled.

    [1] https://opendev.org/starlingx/openstack-armada-app/commit/8254cd31bb1f12eebc48b712b33f75b2fc0aa571

    Test Plan (on AIO-DX with HTTPS enabled):
    PASS - Build openstack-helm package
    PASS - Build stx-openstack-helm-fluxcd package
    PASS - Build stx-openstack helm charts
    PASS - Upload/apply stx-openstack
    PASS - Verify that all pods -- including `cinder-volume-usage-audit` --
           are either "Running" or "Completed"
    PASS - Remove/delete stx-openstack

    Closes-Bug: 2032703

    Change-Id: Ic13c6945cc9e43f9153820297e74623520446fcd
    Signed-off-by: Luan Nunes Utimura <email address hidden>
    (cherry picked from commit be56c15bc043528f4448ae607182d97641fc8766)

tags: added: in-f-antelope
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.