oidc-auth kube-apiserver parameters missing after backup and restore on single node system

[Jerry Sun]

Brief Description
-----------------
Kube-apiserver service parameters are lost after a restore of single node system if the system was backed up with kube-apiserver service parameters. This is because the service parameters can only be applied at runtime. This was discovered during an upgrade and the upgrade was fixed by applying the service parameters. This launchpad tracks the issue to find a more permanent fix for the backup and restore case. We already have a suggestion from the review https://review.opendev.org/c/starlingx/config/+/813257 from David.

Severity
--------
Major: System/Feature is usable but degraded

Steps to Reproduce
------------------
Perform a backup and restore on a single node system with kube-apiserver service parameters (ex oidc-auth)

Expected Behavior
------------------
kube-apiserver service parameters restored

Actual Behavior
----------------
kube-apiserver service parameters are not restored

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
one node system

Branch/Pull Time/Commit
-----------------------
2021-10-21

Workaround
----------
run "system service-parameter-apply kubernetes" after unlocking the controller

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/818136

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/818136
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/3d2eb4e267461614f9c0ae897e058689812d9208
Submitter: "Zuul (22348)"
Branch: master

commit 3d2eb4e267461614f9c0ae897e058689812d9208
Author: Rafael Camargos <email address hidden>
Date: Tue Nov 16 17:13:07 2021 -0300

    Apply apiserver parameters on restore and upgrade

    The issue is manifested by backing-up and restoring a system with
    kube-apiserver parameters applied, e.g.: oidc-auth params. The
    parameters are restored and can be displayed by running
    `service-parameter-list` but aren't getting reapplied back.

    The fix consists in running the `puppet-manifest-apply.sh` script during
    the 'restore-more-data' step of the restore, which is executed both
    during platform restore and upgrade. When applying the manifest the
    kube-apiserver service parameters are updated, similarly to running
    `service-parameter-apply`.

    Test Plan: In an AIO-SX, apply oidc-auth kube-apiserver parameters,
    perform a B&R and an upgrade operation and verify these are properly
    restored.

    PASS: Verify kube-apiserver service parameters are restored after a
    System Backup and Restore
    PASS: Verify kube-apiserver service parameters are restored after a
    System Upgrade

    Closes-Bug: 1948140
    Signed-off-by: Rafael Camargos <email address hidden>
    Change-Id: I9d4823deb0b928acd86c8372fe5306ac8a841518

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/818352
Committed: https://opendev.org/starlingx/config/commit/67ec634c2846e64e13c6e3a82ce33ce402ec2ca6
Submitter: "Zuul (22348)"
Branch: master

commit 67ec634c2846e64e13c6e3a82ce33ce402ec2ca6
Author: Rafael Camargos <email address hidden>
Date: Thu Nov 18 05:56:14 2021 -0300

    Remove sysinv kube-apiserver upgrade params apply

    This can be dropped now that a different fix is submitted on the restore
    ansible playbooks. The new fix was initially proposed for platform
    restore but also fixes the upgrade scenario.

    Test Plan: In an AIO-SX, apply oidc-auth kube-apiserver parameters,
    perform an upgrade operation and verify these are properly restored and
    applied.

    PASS: Verify kube-apiserver service parameters are restored after a
    System Upgrade

    Closes-Bug: 1948140
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/818136
    Signed-off-by: Rafael Camargos <email address hidden>
    Change-Id: I1908e739046d5b215e0f081836fecdfbf8e3a700

[Ghada Khalil (gkhalil)]

Re-opening; the code changes introduced B&R issues on standard systems. The changes are being reverted:
https://review.opendev.org/c/starlingx/config/+/819835
https://review.opendev.org/c/starlingx/ansible-playbooks/+/819834

[Ghada Khalil (gkhalil)]

Moving to stx.7.0 as there is a simple workaround for the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/823714

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/823714
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/4e0b27302fe7b6561982ca7d974eaa1eb5f5c986
Submitter: "Zuul (22348)"
Branch: master

commit 4e0b27302fe7b6561982ca7d974eaa1eb5f5c986
Author: Rafael Camargos <email address hidden>
Date: Thu Jan 6 12:56:12 2022 -0300

    Apply apiserver parameters on restore and upgrade

    The bug being fixed here is manifested by backing-up and restoring a
    system with kube-apiserver parameters applied, e.g.: oidc-auth params.
    The parameters are restored and can be displayed by running
    `service-parameter-list` but aren't getting reapplied back.

    The fix consists in running, during the 'restore-more-data' step of the
    restore:

    - Host and system config data update in order to generate the hieradata
      manifest files
    - The `puppet-manifest-apply.sh` script , which is executed both during
      platform upgrade (AIO-SX only) and restore.

    When applying the manifest the kube-apiserver service parameters are
    updated, similarly to running `service-parameter-apply`.

    Test Plan: Apply oidc-auth kube-apiserver parameters and verify these
    are properly restored after B&R and upgrade

    PASS: Verify kube-apiserver service parameters are restored after a
    AIO-SX System Backup and Restore
    PASS: Verify kube-apiserver service parameters are restored after a
    AIO-SX System Upgrade
    PASS: Verify kube-apiserver service parameters are restored after a
    Standard System Backup and Restore

    Regression: Verify system install

    PASS: Verify AIO-SX System install
    PASS: Verify Standard System install

    Closes-Bug: 1948140
    Signed-off-by: Rafael Camargos <email address hidden>
    Change-Id: I4333024fa711cc7bf515bd600c92b647c232ee30

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/824463

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/824463
Committed: https://opendev.org/starlingx/config/commit/070b03305588d3186a1d17635d94beb4d5024a5b
Submitter: "Zuul (22348)"
Branch: master

commit 070b03305588d3186a1d17635d94beb4d5024a5b
Author: Rafael Camargos <email address hidden>
Date: Thu Nov 18 05:56:14 2021 -0300

    Remove sysinv kube-apiserver upgrade params apply

    With the implementation on
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/823714,
    reapplying the kube-apiserver service parameters on upgrade complete
    step became unnecessary. This change removes the original
    implementation.

    Test Plan: Aply oidc-auth kube-apiserver parameters, perform an upgrade
    operation and verify these are properly restored and applied.

    PASS: Verify kube-apiserver service parameters are restored after a
    SX System Upgrade

    Regression:

    PASS: Verify system install
    PASS: Verify upgrade completes successfully

    Closes-Bug: 1948140
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/823714
    Signed-off-by: Rafael Camargos <email address hidden>
    Change-Id: I1872f65d4aeecda08d64626f7f3b236cfacb1f5e