Secrets associated with cert manager certificates deleted during upgrade

Bug #1985584 reported by Jerry Sun
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Jerry Sun

Bug Description

Brief Description
-----------------
Cert manager certificates secrets are deleted during an upgrade of cert manager.
This causes all certificates to be re-issued during an upgrade

Severity
--------
Major: System/Feature is usable but degraded

Steps to Reproduce
------------------
Perform a system upgrade. Check the creation timestamp of secrets tied to cert manager certificates, they are the same as when cert manager was updated.

Expected Behavior
------------------
Secrets for cert manager certificates are not deleted during cert manager upgrade process

Actual Behavior
----------------
Secrets are deleted, forcing certificates to be renewed

Reproducibility
---------------
Intermittent. Likes to happen more often on busy systems like large DC deployments

System Configuration
--------------------
Two node system

Branch/Pull Time/Commit
-----------------------
2022-08-11

Jerry Sun (jerry-sun-u)
Changed in starlingx:
assignee: nobody → Jerry Sun (jerry-sun-u)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/852991

Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/c/starlingx/config/+/852991
Committed: https://opendev.org/starlingx/config/commit/92375041697cdfb9686ff905f27b7353a31b969a
Submitter: "Zuul (22348)"
Branch: master

commit 92375041697cdfb9686ff905f27b7353a31b969a
Author: Jerry Sun <email address hidden>
Date: Thu Aug 11 15:47:25 2022 -0400

    Do not delete secrets on cert manager upgrade

    Cert manager is configured to delete secrets when the corresponding
    certificate is deleted. As part of data migration during an upgrade,
    we delete and recreate cert manager certificates in the new fluxcd
    version of cert manager. This causes all certificates to be renewed
    due to missing secrets when cert manager is upgraded. This commit
    adds a system override to NOT delete secrets when a certificate is
    deleted. It also removes the upstream documentation step of deleting
    CRDs, since the CRDs are already handled by helm.

    Test Cases:

    PASS: Upgrade on a setup that pretty reliably reproduces the issue.
          The cert manager certificate secrets were not recreated.
    PASS: Manually execute the steps in the script slowly. Without
          the change, secrets were recreated. With the change, they
          were not.

    Change-Id: I55715fc4e6f4d2eb9c2e6429801c2b2a2044e122
    Closes-Bug: 1985584
    Signed-off-by: Jerry Sun <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
tags: added: stx.8.0 stx.security
Changed in starlingx:
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.