Run upgrade manifest fails during upgrade-activate with missing update-k8s-feature-gates.sh on Debian

[Kaustubh Dhokte]

Brief Description
-----------------
During upgrade-activate, upgrade manifest execution on Debian fails.

Severity
--------
Major

Steps to Reproduce
------------------
Start CentOS to Debian AIO-DX upgrade
Upgrade and unlock controller-1
Swact
Upgrade and unlock controller-0
Run upgrade-activate

Expected Behavior
------------------
upgrade-activate should succeed.

Actual Behavior
----------------
upgrade-activate fails.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
AIO-SX

Timestamp/Logs
--------------
2022-08-15T08:03:32.767 Error: 2022-08-15 08:03:30 +0000 Could not find command '/usr/local/sbin/update-k8s-feature-gates.sh'
2022-08-15T08:03:32.770 /usr/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:21:in `checkexe'
2022-08-15T08:03:32.772 /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:39:in `run'
2022-08-15T08:03:32.775 /usr/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:45:in `run'
2022-08-15T08:03:32.777 /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:130:in `block in sync'
2022-08-15T08:03:32.779 /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `times'
2022-08-15T08:03:32.783 /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `sync'
2022-08-15T08:03:32.786 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync'
2022-08-15T08:03:32.788 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
2022-08-15T08:03:32.790 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
2022-08-15T08:03:32.793 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
2022-08-15T08:03:32.796 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
2022-08-15T08:03:32.798 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
2022-08-15T08:03:32.801 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:259:in `apply'
2022-08-15T08:03:32.805 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:279:in `eval_resource'
2022-08-15T08:03:32.808 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `call'
2022-08-15T08:03:32.810 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block (2 levels) in evaluate'
2022-08-15T08:03:32.813 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2022-08-15T08:03:32.815 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2022-08-15T08:03:32.818 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2022-08-15T08:03:32.820 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block in evaluate'
2022-08-15T08:03:32.823 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
2022-08-15T08:03:32.825 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:173:in `evaluate'
2022-08-15T08:03:32.827 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
2022-08-15T08:03:32.830 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2022-08-15T08:03:32.832 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2022-08-15T08:03:32.835 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2022-08-15T08:03:32.837 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
2022-08-15T08:03:32.840 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
2022-08-15T08:03:32.842 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
2022-08-15T08:03:32.845 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
2022-08-15T08:03:32.847 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:190:in `block (2 levels) in apply_catalog'
2022-08-15T08:03:32.850 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2022-08-15T08:03:32.852 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2022-08-15T08:03:32.855 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2022-08-15T08:03:32.857 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:189:in `block in apply_catalog'
2022-08-15T08:03:32.860 /usr/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
2022-08-15T08:03:32.862 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2022-08-15T08:03:32.865 /usr/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
2022-08-15T08:03:32.867 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:188:in `apply_catalog'
2022-08-15T08:03:32.870 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:390:in `run_internal'
2022-08-15T08:03:32.873 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:238:in `block in run'
2022-08-15T08:03:32.875 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2022-08-15T08:03:32.878 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2022-08-15T08:03:32.880 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:215:in `run'
2022-08-15T08:03:32.883 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:355:in `apply_catalog'
2022-08-15T08:03:32.887 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
2022-08-15T08:03:32.890 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2022-08-15T08:03:32.893 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2022-08-15T08:03:32.896 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
2022-08-15T08:03:32.898 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2022-08-15T08:03:32.900 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2022-08-15T08:03:32.903 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
2022-08-15T08:03:32.905 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
2022-08-15T08:03:32.907 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
2022-08-15T08:03:32.910 /usr/lib/ruby/vendor_ruby/puppet/util.rb:710:in `exit_on_fail'
2022-08-15T08:03:32.913 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
2022-08-15T08:03:32.915 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
2022-08-15T08:03:32.918 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
2022-08-15T08:03:32.921 /usr/bin/puppet:11:in `<main>'
2022-08-15T08:03:32.923 Error: 2022-08-15 08:03:30 +0000 /Stage[main]/Platform::Kubernetes::Master::Update_kubeadm_feature_gates/Exec[update kubeadm feature-gates]/returns: change from 'notrun' to ['0'] failed: Could not find command '/usr/local/sbin/update-k8s-feature-gates.sh'

Test Activity

Workaround
----------
This script was created on https://review.opendev.org/c/starlingx/integ/+/845654 and by the comments on it this is a known issue, the script must now be created on Debian to support upgrades. Working around this by creating the script manually on both controllers allowed the manifest to complete successfully.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/859338

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/859338
Committed: https://opendev.org/starlingx/stx-puppet/commit/285692c61cdfb44036b7d0f4edc9a6cc0150a149
Submitter: "Zuul (22348)"
Branch: master

commit 285692c61cdfb44036b7d0f4edc9a6cc0150a149
Author: Kaustubh Dhokte <email address hidden>
Date: Mon Sep 26 14:06:23 2022 -0400

    update-k8s-feature-gates script presence check

    The change https://opendev.org/starlingx/integ/commit/a6a5349d025487672fe818aae36a2020a9f9f08c
    adds a script that removes deprecated kube-apiserver feature
    gates. The script is run during platform upgrade (upgrade activate phase).
    This puppet class https://opendev.org/starlingx/stx-puppet/commit/1cdfd7828622b88318058ae4b13b629fc1d94be9
    invokes it .

    The script is packaged in kubernetes-1.22.5 package.
    The package is currently only supported on CentOS and not on
    Debian. As a result, the script is missing on Debian and
    platform upgrades on Debian are failing.

    This change provides a quick temporary fix to upgrade failures
    before the script is made available for Debian.
    The change adds a check if the script is present before it is
    executed, thus avoiding upgrade failures on Debian and unblocks
    upgrade testing.

    Testing:
    This is tested on CentOS as Debian upgrades are still WIP.
    On AIO-SX:
    PASS: Test upgrade-activate by manually deleting the script.
          upgrade-activate completes successfully.

    Partial-Bug: 1990880

    Signed-off-by: Kaustubh Dhokte <email address hidden>
    Change-Id: Ie35e54e207b2e17b4e906cd6f1cccaa4818a0a2b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/864427
Committed: https://opendev.org/starlingx/integ/commit/6e7736059ac927860951c876ad2693c03d9bd51b
Submitter: "Zuul (22348)"
Branch: master

commit 6e7736059ac927860951c876ad2693c03d9bd51b
Author: Kaustubh Dhokte <email address hidden>
Date: Mon Nov 14 16:28:47 2022 +0000

    update feature-gates for specific k8s version

    This change modifies upgrade_k8s_config.sh to support updating
    k8s feature-gates for different k8s versions. With every k8s release,
    default values of some feature-gate are changed and usage of some
    feature-gates often gets deprecated.

    The script runs during each k8s control plane upgrade before
    upgrading first master. It modifies kubeadm-config configmap
    with features-gates as required for the specific k8s version
    we are upgrading to.

    The set of changes here, https://opendev.org/starlingx/integ/commit/a6a5349d025487672fe818aae36a2020a9f9f08c
    (k8s-1.22.5: remove feature-gates), https://opendev.org/starlingx/stx-puppet/commit/1cdfd7828622b88318058ae4b13b629fc1d94be9
    (Add a puppet class to support k8s feature-gate update), and https://opendev.org/starlingx/config/commit/cc3cdbd6474e574a20bc71b35cb6875e359875aa
    (apply feature-gate update during upgrade-activate) were added for
    stx 6.0 to stx 7.0 upgrade (CentOS) for changes in feature-gates with
    respect to k8s 1.22. Instead of adding that script to Debian and
    maintaining two different scripts, going forward we can maintain this
    single script to accommodate any change in feature-gates
    (or any other config in kubeadm-config) with respect to the specific
    k8s version we are upgrading to.

    Test Plan:
    PASS: K8s upgrade 1.21.8 to 1.22.5
    PASS: k8s upgrade 1.23.1 to 1.24.4
    PASS: shellcheck run
    PASS: replace_configmap function was unit tested separately.

    Closes-Bug: 1996546
    Closes-Bug: 1990880

    Signed-off-by: Kaustubh Dhokte <email address hidden>
    Change-Id: Ib693d7892aee2da91d612789b64ff38a65da5ccb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/867143

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/864429
Committed: https://opendev.org/starlingx/config/commit/5a9fe598e6be456d9b306100ad44e454891e2471
Submitter: "Zuul (22348)"
Branch: master

commit 5a9fe598e6be456d9b306100ad44e454891e2471
Author: Kaustubh Dhokte <email address hidden>
Date: Mon Nov 14 16:51:30 2022 +0000

    remove update-kubeadm-feature-gates manifest run

    This change removes the execution of the manifest
    update_kubeadm_feature_gates during platform upgrades.

    The set of changes here, https://opendev.org/starlingx/integ/commit/a6a5349d025487672fe818aae36a2020a9f9f08c
    (k8s-1.22.5: remove feature-gates), https://opendev.org/starlingx/stx-puppet/commit/1cdfd7828622b88318058ae4b13b629fc1d94be9
    (Add a puppet class to support k8s feature-gate update), and https://opendev.org/starlingx/config/commit/cc3cdbd6474e574a20bc71b35cb6875e359875aa
    (apply feature-gate update during upgrade-activate) were added for
    stx 6.0 to stx 7.0 upgrade (CentOS) for changes in feature-gates with
    respect to k8s 1.22.
    Updating feature gates as required per k8s version is now handled
    at k8s upgrade in a single script (upgrade-k8s-config) starting
    from this change
    https://opendev.org/starlingx/integ/commit/6e7736059ac927860951c876ad2693c03d9bd51b
    (update feature-gates for specific k8s version).
    Hence, update-kubeadm-feature-gates manifest is no longer required.

    Test Plan:
    PASS: AIO-SX stx7.0 to stx8.0 platform upgrade successful.

    Closes-Bug: 1990880

    Signed-off-by: Kaustubh Dhokte <email address hidden>
    Change-Id: I08904a54e8295c99e24113efa73c9e6590e1b8a9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/864432
Committed: https://opendev.org/starlingx/stx-puppet/commit/c8a32f609904a64063f5f74f1285bddfe12939d0
Submitter: "Zuul (22348)"
Branch: master

commit c8a32f609904a64063f5f74f1285bddfe12939d0
Author: Kaustubh Dhokte <email address hidden>
Date: Mon Nov 14 16:58:56 2022 +0000

    remove puppet class update_kubeadm_feature_gates

    This change removes puppet class update_kubeadm_feature_gates
    from kubernetes.pp.

    The set of changes here,
    https://opendev.org/starlingx/integ/commit/a6a5349d025487672fe818aae36a2020a9f9f08c
    (k8s-1.22.5: remove feature-gates),

    https://opendev.org/starlingx/stx-puppet/commit/1cdfd7828622b88318058ae4b13b629fc1d94be9
    (Add a puppet class to support k8s feature-gate update), and

    https://opendev.org/starlingx/config/commit/cc3cdbd6474e574a20bc71b35cb6875e359875aa
    (apply feature-gate update during upgrade-activate) were added for
    stx 6.0 to stx 7.0 upgrade (CentOS) for changes in feature-gates with
    respect to k8s 1.22.

    Updating feature gates as required per k8s version is now handled
    at k8s upgrade in a single script (upgrade-k8s-config) starting
    from this change:
    https://opendev.org/starlingx/integ/commit/6e7736059ac927860951c876ad2693c03d9bd51b
    (update feature-gates for specific k8s version).

    Hence, this puppet class is no longer required.

    Depends-On: https://review.opendev.org/c/starlingx/config/+/864429

    Test Plan:
    PASS: AIO-SX stx7.0 to stx8.0 platform upgrade successful.

    Closes-Bug: 1990880

    Signed-off-by: Kaustubh Dhokte <email address hidden>
    Change-Id: Idde35458bed976e483e825bef5477db085ec124e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/867143
Committed: https://opendev.org/starlingx/integ/commit/c71a3ea4243f46656313e3afc0b5c235d99e265c
Submitter: "Zuul (22348)"
Branch: master

commit c71a3ea4243f46656313e3afc0b5c235d99e265c
Author: Kaustubh Dhokte <email address hidden>
Date: Fri Dec 9 20:28:33 2022 +0000

    remove script update-k8s-feature-gates

    The set of changes here,
    (k8s-1.22.5: remove feature-gates)
    https://opendev.org/starlingx/integ/commit/a6a5349d025487672fe818aae36a2020a9f9f08c,

    (Add a puppet class to support k8s feature-gate update)
    https://opendev.org/starlingx/stx-puppet/commit/1cdfd7828622b88318058ae4b13b629fc1d94be9

    and

    (apply feature-gate update during upgrade-activate)
    https://opendev.org/starlingx/config/commit/cc3cdbd6474e574a20bc71b35cb6875e359875aa

    were added for stx 6.0 to stx 7.0 upgrade (CentOS) for changes in
    feature-gates with respect to k8s 1.22.

    Updating feature gates as required per k8s version is now handled
    at k8s upgrade in a single script (upgrade-k8s-config) starting
    from this change:
    (update feature-gates for specific k8s version)
    https://opendev.org/starlingx/integ/commit/6e7736059ac927860951c876ad2693c03d9bd51b

    So, update-k8s-feature-gates script is no longer required.

    Closes-Bug: 1990880

    Test Plan:
    On AIO-SX
    PASS: Check script is not present after stx7.0 to stx8.0 platform
          upgrade.
    PASS: AIO-SX stx7.0 to stx8.0 platform upgrade successful.

    Signed-off-by: Kaustubh Dhokte <email address hidden>
    Change-Id: Id85628a677877048c6c5aa1747a33fa8c72056a3