kubernetes certificates renewal failure on controller-1 after initial install

[Danilo]

Brief Description
-----------------
Alarm 250.003, kubernetes certificates renewal failed is raised for controller-1.

Severity
--------
Major: System is usable but degraded

Steps to Reproduce
------------------
Install STX and wait for k8s certificates to expire on controller-1

Expected Behavior
------------------
all k8s certificates should get renewed without issues

Actual Behavior
----------------
k8s certificates fail to be renewed on controller-1

Reproducibility
---------------
Seen once

System Configuration
--------------------
DX

Branch/Pull Time/Commit
-----------------------
STX master 2023-07-31 06:00:00
STX-O master 2023-07-31 16:01:00

Last Pass
---------
2023_07_06-06_00_00

Timestamp/Logs
--------------
+----------+-----------------------------------------+-------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+-----------------------------------------+-------------------+----------+----------------+
| 250.003 | Kubernetes certificates renewal failed. | host=controller-1 | major | 2023-08-02T00: |
| | | | | 10:01.424863 |
| | | | | |
+----------+-----------------------------------------+-------------------+----------+----------------+

Test Activity
-------------
Sanity Testing

Workaround
----------
N/A

[Ghada Khalil (gkhalil)]

This is resulting in stx-openstack sanity failing; this needs to be triaged by the security team.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/890442
Committed: https://opendev.org/starlingx/config/commit/8170a07bca3e7e5c425240174c89cc1354d96696
Submitter: "Zuul (22348)"
Branch: master

commit 8170a07bca3e7e5c425240174c89cc1354d96696
Author: Rei Oliveira <email address hidden>
Date: Thu Aug 3 14:07:17 2023 -0300

    Fix error when running k8s cert rotation on c1

    Review https://review.opendev.org/c/starlingx/config/+/884627
    introduced parameter --config /etc/kubernetes/kubeadm.yaml for
    kubeadm certs check-expiration, which generates a better output
    without warnings, that are print when run without.

    File /etc/kubernetes/kubeadm.yaml, however, is not present on
    controller-1 so this is not a good solution. This commit removes the
    usage of the parameter so that the script can work on both environments.

    This script runs as a cron job only on controller nodes.

    PASS: Trigger execution of script kube-cert-rotation.sh on c0, verify
          that it runs without error. Run 'fm alarm-list' and verify no
          'Kubernetes certificates renewal failed are present'
    PASS: Trigger execution of script kube-cert-rotation.sh on c1, verify
          that it runs without error. Run 'fm alarm-list' and verify no
          'Kubernetes certificates renewal failed are present'

    Closes-Bug: 2029506
    Closes-Bug: 2029378

    Signed-off-by: Rei Oliveira <email address hidden>
    Change-Id: I1ea458163dc4250a3bc3550eaaa68d314224023e