Leaf certificates are renewed based on invalid CA certificates

[Reinildes Oliveira]

Brief Description
------------------------------------
Leaf certificates are renewed based on invalid CA certificates

Error scenario:

subcloud rehome with with expired certificates

Error condition:

If we leave the system with expired certificates for more than 1 day, /usr/bin/kube-cert-rotation.sh cron job execution that renews the leaf certificates based on CA certificates that, in this case, are in invalid state. This will make the system to interpret that leaf certificates were already renewed and are in valid state.
How to verify the error condition?
What is the side effects?

Severity
------------------------------------

<Critical: System/Feature is not usable after the defect>

Steps to Reproduce

Deploy systemcontroller A and subclouds with HW clock set to current date
Deploy systemcontroller B with clock set to 11 years ahead
Update subclouds HW clock to 11 years ahead
It will make the subclouds certificates to be expired
Leave the system in this state for at least one day
Rehome subclouds to systemcontroller B

Expected Behavior
------------------------------------

The subclouds should be rehomed successfully and all certificates should be in valid state

Actual Behavior
------------------------------------

The subcloud is rehomed successfully, however leaf certificates are not in valid state

Reproducibility
------------------------------------

100% reproducible

System Configuration
------------------------------------

DC

Load info (eg: 2022-03-10_20-00-07)

// code placeholder

Last Pass
------------------------------------

new test scenario.

Alarms
------------------------------------
no alarms

Test Activity
------------------------------------
Regression Testing

Workaround
------------------------------------
not available

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/914684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/914684
Committed: https://opendev.org/starlingx/config/commit/01a5ea0843bd25a422993ff267567609f67351a5
Submitter: "Zuul (22348)"
Branch: master

commit 01a5ea0843bd25a422993ff267567609f67351a5
Author: Rei Oliveira <email address hidden>
Date: Thu Mar 28 14:28:34 2024 -0300

    First check Root CAs on kube-cert-rotation.sh

    As of now, the script only verifies the validity of leaf certificates
    and, if expired, will regenerate them based on K8s/etcd Root CAs.
    It doesn't account for the possibility of Root CAs being expired.
    It will generate leaf certificates based on Root CAs, even if said
    Root CAs are expired.

    This change fixes that behaviour by first checking validity of
    Root CAs and only allowing leaf certificate renewal if RCAs are
    valid.

    Test plan:

    PASS: Cause Root CAs to expire, run kube-cert-rotation.sh script
          and verify that it fails with an error saying Root CAs are
          expired and leaf certificates are not renewed.
    PASS: Ensure to have valid Root CAs, cause leaf certificates
          to expire, run kube-cert-rotation.sh and verify that the
          script executes normally and is able to renew
          the leaf certificates.

    Closes-Bug: 2059708

    Signed-off-by: Rei Oliveira <email address hidden>
    Change-Id: I98dfd8d1417754f3c723d8ddd52a856785ffc83b