CrashLoopBackoff pods not being detected by kube rootca health check

[Victor Romano]

Brief Description
-----------------
Kube rootca update continue to run if there's a pod in CrashLoopBackoff state.

Severity
--------
Minor

Steps to Reproduce
------------------
1. sw-manager kube-rootca-update-strategy create
2. sw-manager kube-rootca-update-strategy apply

Expected Behavior
------------------
Apply should fail because of failed pod

Actual Behavior
----------------
Apply continue

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Any system (tested on a duplex system)

Branch/Pull Time/Commit
-----------------------
NA

Last Pass
---------
NA

Timestamp/Logs
--------------
NA

Test Activity
-------------
Developer testing

Workaround
----------
NA

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/912770

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/912770
Committed: https://opendev.org/starlingx/config/commit/d807f868d6eb7378d3c7a7a9dbfad2bd9a572627
Submitter: "Zuul (22348)"
Branch: master

commit d807f868d6eb7378d3c7a7a9dbfad2bd9a572627
Author: Victor Romano <email address hidden>
Date: Wed Mar 13 09:06:49 2024 -0300

    Fix failed pods not being detected by rootca health check

    On the health check prior to rootca update, there was a bug that
    prevented CrashLoopBackoff pods being detected as unhealthy. This is
    because the pods are in phase "Running", but the status of the
    container itself is "ready: false". This commit adds an additional
    check to "Running" pods so if any container inside it is not ready,
    the pod will be deemed unhealthy.

    Test plan:
      - PASS: Attempt to perform a rootca update with a pod in
              CrashloopBackoff state. Verify the update is not possible
              and the health check fails with the pod being show as
              unhealthy is "system health-query-kube-upgrade --rootca"
      - PASS: Verify the rootca update is possible if no pods are in
              CrashloopBackoff state.

    Closes-Bug: 2057779

    Change-Id: I115b6621df11516db2279fe6bc96452d27975c50
    Signed-off-by: Victor Romano <email address hidden>