kubernetes apiserver certificate needs rotation

Bug #1838659 reported by David Sullivan on 2019-08-01
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
High
Mingyuan Qi

Bug Description

Brief Description
-----------------
When the apiserver/apiserver-kubelet-client certficiates expire access to the kubernetes api server is lost.

Severity
--------
Critical

Steps to Reproduce
------------------
Install and configure an AIO-SX system
Verify the expiry of the apiserver certificate with `openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt`
Set the date of the system to later than the certificate expiry

Expected Behavior
------------------
Sometime before the expiry a new certificate needs to be generated.

Actual Behavior
----------------
Kubelet can not connect to the apiserver as the certificate is no longer valid.

Reproducibility
---------------
100%

System Configuration
--------------------
All systems

Branch/Pull Time/Commit
-----------------------
20190728T233000Z

Last Pass
---------
NA

Timestamp/Logs
--------------
controller-0:/home/sysadmin# openssl x509 -text -noout -in /etc/kubernetes/pki/apiserver.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 5878483830693726211 (0x519491e602608803)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Jul 25 21:06:05 2019 GMT
            Not After : Jul 24 21:06:05 2020 GMT
        Subject: CN=kube-apiserver
...

controller-0:/home/sysadmin# date
Fri Jul 24 21:00:02 UTC 2020
controller-0:/home/sysadmin# date
Fri Jul 24 21:07:18 UTC 2020
controller-0:/home/sysadmin# kubectl get pods -n kube-system
Unable to connect to the server: x509: certificate has expired or is not yet valid

Test Activity
-------------
Developer Testing

Frank Miller (sensfan22) on 2019-08-02
tags: added: stx.2.0 stx.config stx.containers
Changed in starlingx:
status: New → Triaged
importance: Undecided → High
Cindy Xie (xxie1) on 2019-08-03
Changed in starlingx:
assignee: nobody → Mingyuan Qi (myqi)
Mingyuan Qi (myqi) wrote :

Successfully updated apiserver, controller-manager, scheduler, kubelet, kubectl certificate manually. Figuring out an approach to automatically detect certificate expiration.

Fix proposed to branch: master
Review: https://review.opendev.org/692276

Changed in starlingx:
status: Triaged → In Progress
Ghada Khalil (gkhalil) wrote :

Adding the stx.3.0 release tag explicitly as this fix is needed for that release as well

tags: added: stx.3.0

Reviewed: https://review.opendev.org/696224
Committed: https://git.openstack.org/cgit/starlingx/fault/commit/?id=bc6796cdc7995575409fd47b4d4f9a8b31f91ebf
Submitter: Zuul
Branch: master

commit bc6796cdc7995575409fd47b4d4f9a8b31f91ebf
Author: Mingyuan Qi <email address hidden>
Date: Wed Nov 27 02:41:40 2019 +0000

    Add an alarm for k8s certificate rotation

    This alarm will be raised if the automatic k8s cert rotation failed
    The k8s cert automatic rotation is implemented in commit:
    https://review.opendev.org/#/c/692276/

    Change-Id: Idddd6ae7b83bc40b805e85004994c48cd801ee75
    Partial-Bug: 1838659
    Signed-off-by: Mingyuan Qi <email address hidden>

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers