Second master node installed with unsupported method

Bug #1846829 reported by Bart Wensley
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
David Sullivan

Bug Description

Brief Description
-----------------
When we originally implemented the deployment of the kubernetes HA cluster using kubeadm, the instructions for the second master node were to copy certificate files from the first master node and then run the "kubeadm init" command with the same yaml file used on the first master node. However, in a subsequent release, the instructions (see https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability) changed to use the "kubeadm join" with a --control-plane option instead. This approach is simpler and the approach we are currently using is no longer supported (although it still seems to be working).

Severity
--------
Minor - although I expect the method we are using will no longer work at some point

Steps to Reproduce
------------------
Install a system with two controllers

Expected Behavior
------------------
We should be using the supported kubadm instructions.

Actual Behavior
----------------
We are following obsolete instructions.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Any system with two controller hosts

Branch/Pull Time/Commit
-----------------------
Any

Last Pass
---------
N/A

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer Testing

Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as stx.3.0 / medium priority - prefer to align on the supported method for the stx.3.0 release.

tags: added: stx.containers
tags: added: stx.3.0
Changed in starlingx:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Bart Wensley (bartwensley)
Changed in starlingx:
assignee: Bart Wensley (bartwensley) → David Sullivan (dsullivanwr)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

As per agreement with the community, moving unresolved medium priority bugs (< 100 days OR recently reproduced) from stx.3.0 to stx.4.0

tags: added: stx.4.0
removed: stx.3.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/702797

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (master)

Fix proposed to branch: master
Review: https://review.opendev.org/702798

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (master)

Fix proposed to branch: master
Review: https://review.opendev.org/703911

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/702797
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=3c1eb398c8294465b05b45bd35a4b5a359c170b4
Submitter: Zuul
Branch: master

commit 3c1eb398c8294465b05b45bd35a4b5a359c170b4
Author: David Sullivan <email address hidden>
Date: Wed Jan 15 23:47:59 2020 -0500

    Install secondary controller nodes with kubeadm join

    Kubeadm init is no longer supported for installing secondary nodes in an
    HA kubernetes cluster. kubeadm join with the --controller-plane option
    should be used.

    Change-Id: I57d2d01335739fb3dce5a9c4cd749b99eafbf5ee
    Partial-Bug: 1846829
    Signed-off-by: David Sullivan <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (master)

Reviewed: https://review.opendev.org/702798
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=83775d38804fb665af518127051b37a1daf31e36
Submitter: Zuul
Branch: master

commit 83775d38804fb665af518127051b37a1daf31e36
Author: David Sullivan <email address hidden>
Date: Wed Jan 15 23:50:23 2020 -0500

    Install secondary controller nodes with kubeadm join

    Kubeadm init is no longer supported for installing secondary nodes in an
    HA kubernetes cluster. kubeadm join with the --controller-plane option
    should be used.

    Change-Id: I21a30b9e871d05c59a19e33a9d278f0217682da6
    Closes-Bug: 1846829
    Depends-On: https://review.opendev.org/702797
    Signed-off-by: David Sullivan <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/703911
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=92ca122652733805b62fc16940861ca4e83e2bb1
Submitter: Zuul
Branch: master

commit 92ca122652733805b62fc16940861ca4e83e2bb1
Author: David Sullivan <email address hidden>
Date: Wed Jan 22 21:33:19 2020 -0500

    Install secondary controller nodes with kubeadm join

    Kubeadm init is no longer supported for installing secondary nodes in an
    HA kubernetes cluster. kubeadm join with the --controller-plane option
    should be used.

    Change-Id: I64aaf02b09053608c884149d73bc1a3f2b62d98a
    Partial-Bug: 1846829
    Depends-On: https://review.opendev.org/702797
    Signed-off-by: David Sullivan <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/705831

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/705837

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/705852

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (f/centos8)
Download full text (8.2 KiB)

Reviewed: https://review.opendev.org/705831
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=6670caf7ceda5fe0dc46f2f82033b68abf00ed5e
Submitter: Zuul
Branch: f/centos8

commit bf8d081a95a9b1776964960a6d9089b1449f2c58
Author: Angie Wang <email address hidden>
Date: Thu Jan 30 17:57:05 2020 -0500

    Support k8s networking upgrade based on k8s version

    Update to support a set of k8s networking templates
    based on kubernetes release. The kubernetes version
    needs to be passed to the ansible playbook
    k8s-networking-upgrade.yml to determine which set
    of networking manifests should be applied for the
    current kubernetes.

    Story: 2006781
    Task: 37584
    Change-Id: I3a0b9f56608ddb1323b36f9ecedb8a5488c222c9
    Signed-off-by: Angie Wang <email address hidden>

commit 2b0cd43e5fa75628d8eff78be7045ba4fc82d980
Author: Jerry Sun <email address hidden>
Date: Thu Dec 19 13:22:50 2019 -0500

    Add Dex parameters to ansible bootstrap

    Add oidc_groups_claim as a new parameters for ansible
    config. We now have 2 valid configs: the previous 3 parameters
    for a microsoft azure authentication deployment, and the previous
    3 in addition to oidc_groups_claim for a dex authentication
    deployment.

    Story: 2006711
    Task: 37850
    Change-Id: I265d2f7872eb31e2b295eeff6a3165543673497c
    Depends-On: https://review.opendev.org/702798
    Signed-off-by: Jerry Sun <email address hidden>

commit 92ca122652733805b62fc16940861ca4e83e2bb1
Author: David Sullivan <email address hidden>
Date: Wed Jan 22 21:33:19 2020 -0500

    Install secondary controller nodes with kubeadm join

    Kubeadm init is no longer supported for installing secondary nodes in an
    HA kubernetes cluster. kubeadm join with the --controller-plane option
    should be used.

    Change-Id: I64aaf02b09053608c884149d73bc1a3f2b62d98a
    Partial-Bug: 1846829
    Depends-On: https://review.opendev.org/702797
    Signed-off-by: David Sullivan <email address hidden>

commit 393379bd7671aeec5e9852679a69bdc29577426a
Author: Angie Wang <email address hidden>
Date: Tue Jan 28 14:01:10 2020 -0500

    Fix the image download failure on IPv6 system

    "crictl pull" failed to pull images on IPv6 system with
    proxy setting since Containerd doesn't work with the
    NO_PROXY environment variable that has IPv6 addresses
    with square brackets. This commit updates to strip out
    the square brackets from NO_PROXY environment variable.

    Verified on both IPv4 and IPv6 labs.

    Change-Id: I70bd00439b2cc39d2b25dd62746994a524be4998
    Partial-Bug: 1859835
    Signed-off-by: Angie Wang <email address hidden>

commit 792ea357e2b6d2bd23b441aa1657e0dc46f7ef5d
Author: Jim Somerville <email address hidden>
Date: Mon Jan 27 16:08:48 2020 -0500

    Security: Add nospectre_v1 to the default setting

    Most of the v1 mitigation is baked into the kernel and not
    optional. The swapgs barriers are, however, optional.
    They have a negative performance impact so we disable them
    by using the nospectre_v1 kernel bootarg.

    C...

Read more...

tags: added: in-f-centos8
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (f/centos8)
Download full text (35.0 KiB)

Reviewed: https://review.opendev.org/705837
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=8ac6ec70cb8a787a274fd7227eb34d2b7bcd5f5b
Submitter: Zuul
Branch: f/centos8

commit 7995dd436954b92f1c4e3f760a7609af670c84c8
Author: Jessica Castelino <email address hidden>
Date: Mon Feb 3 12:07:26 2020 -0500

    Unit test cases for helm charts

    Test cases added for API endpoints used by:
     1. helm-override-delete
     2. helm-override-show
     3. helm-override-list
     4. helm-override-update
     5. helm-chart-attribute-modify

    Story: 2007082
    Task: 38012
    Change-Id: I86763496bb41084c006f2486702c3b15bde039d2
    Signed-off-by: Jessica Castelino <email address hidden>

commit 7e2fda010299f7305b630d6db97bbe1e169a38b1
Author: Angie Wang <email address hidden>
Date: Wed Jan 29 21:18:18 2020 -0500

    Finish kubernetes networking upgrade support

    The commit completes the RPC kube_upgrade_networking
    in sysinv-conductor to run ansible playbook
    upgrade-k8s-networking.yml to upgrade networking pods
    and also updates the networking upgrade function called
    as part of sysinv-conductor startup to provide a current
    kubernetes version when running the upgrade playbook.

    The second control plane upgrade can only be performed
    after the networking upgrade is done, fix the semantic
    check in sysinv api.

    Change-Id: I8dcf5a2baedfaefb0a7ca037eb47bf7cacd686f8
    Story: 2006781
    Task: 37584
    Depends-On: https://review.opendev.org/#/c/705310/
    Signed-off-by: Angie Wang <email address hidden>

commit 52c37a35d2cd62fa1cc1933765c76c1ba8616864
Author: Jerry Sun <email address hidden>
Date: Fri Jan 31 16:10:25 2020 -0500

    Add Unit Tests for Dex Sysinv Changes

    Add unit tests for the dex helm chart changes under the same story
    and task

    Story: 2006711
    Task: 37857

    Depends-On: https://review.opendev.org/#/c/705297/

    Change-Id: I3a0e1c490e56188adfbd614fd6ebb21bfdddf49e
    Signed-off-by: Jerry Sun <email address hidden>

commit 144587a6ac9fc48b9249be99abadd35dfa49e7a7
Author: Teresa Ho <email address hidden>
Date: Fri Jan 31 15:35:04 2020 -0500

    Tox tests for OIDC client helm overrides

    Added some tox tests for OIDC client helm overrides.

    Story: 2006711
    Task: 38481

    Change-Id: If4aeaf0010c7076d1d83bacd00d6fd0122d4ffad
    Signed-off-by: Teresa Ho <email address hidden>

commit 763ddeadd4e83af6cebf752d693ee3e7d3b005b1
Author: Thomas Gao <email address hidden>
Date: Wed Jan 29 16:30:40 2020 -0500

    Fixed errors in address deletion

    Allowed address deletion despite missing associated interface or host.

    Enabled relevant unit test.

    Closes-Bug: 1860186

    Change-Id: Ie6e6358aa75091e92914a8b581b4d5203a596f56
    Signed-off-by: Thomas Gao <email address hidden>

commit 61463608169e75601b8a4f9db7c98190788d6f6a
Author: Thomas Gao <email address hidden>
Date: Tue Jan 28 15:32:58 2020 -0500

    Fixed broken sysinv address get-all api call

    Removed unexpected keyword argument that caused the error....

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (f/centos8)
Download full text (9.5 KiB)

Reviewed: https://review.opendev.org/705852
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=e1f095eb112f76a133734a17f01afeb9828ebaf2
Submitter: Zuul
Branch: f/centos8

commit fc7b9b3d8d811fd50427b584dae5b7488947bb03
Author: Angie Wang <email address hidden>
Date: Tue Jan 28 13:57:52 2020 -0500

    Fix the image download failure on IPv6 system

    "crictl pull" failed to pull images on IPv6 system with
    proxy setting since Containerd doesn't work with the
    NO_PROXY environment variable that has IPv6 addresses
    with square brackets. This commit updates to strip out
    the square brackets from NO_PROXY environment variable.

    Change-Id: I6bb5ad0379f576f66d77a90dfdca94f5e0f28f0c
    Closes-Bug: 1859835
    Signed-off-by: Angie Wang <email address hidden>

commit 950670ac1f0bfaa43e29eeb3ffda71a94de66520
Author: Jim Somerville <email address hidden>
Date: Mon Jan 27 17:09:52 2020 -0500

    Security: Add nospectre_v1 to the security params

    Most of the v1 mitigation is baked into the kernel and not
    optional. The swapgs barriers are, however, optional.
    They have a negative performance impact so we disable them
    by using the nospectre_v1 kernel bootarg.

    Partial-Bug: 1860193
    Depends-On: https://review.opendev.org/#/c/704406
    Change-Id: Iaa11ba3f430fc064ebda679cf290474d3be413da
    Signed-off-by: Jim Somerville <email address hidden>

commit 83775d38804fb665af518127051b37a1daf31e36
Author: David Sullivan <email address hidden>
Date: Wed Jan 15 23:50:23 2020 -0500

    Install secondary controller nodes with kubeadm join

    Kubeadm init is no longer supported for installing secondary nodes in an
    HA kubernetes cluster. kubeadm join with the --controller-plane option
    should be used.

    Change-Id: I21a30b9e871d05c59a19e33a9d278f0217682da6
    Closes-Bug: 1846829
    Depends-On: https://review.opendev.org/702797
    Signed-off-by: David Sullivan <email address hidden>

commit c94fa4a0174b96e0716d39bbea7e6fbbbee415a9
Author: Shuicheng Lin <email address hidden>
Date: Thu Jan 23 02:45:31 2020 +0800

    Fix duplex system controller-1 fail to boot after unlock

    It is due to controller-1 doesn't have /opt/platform/config folder.
    And cause puppet failure due to using non-exist file as source.
    Restrict the code for worker node only, since controller node
    already has ca cert in the ssl folder.

    Test:
    Pass simplex/duplex/multi node deployment with vm created.

    Closes-Bug: 1860529
    Change-Id: I808ee15e5c78ebead114219d0ec428fb45cc9128
    Signed-off-by: Shuicheng Lin <email address hidden>

commit 27f167eb14a04bc67ecca59af3b617c115522101
Author: Angie Wang <email address hidden>
Date: Wed Jan 15 16:15:26 2020 -0500

    Remove puppet-manifests code made obsolete by ansible

    As a result of switch to Ansible, remove the obsolete erb
    templates and remove the dependency of is_initial_config_primary
    facter.

    Change-Id: I4ca6525f01a37da971dc66a11ee99ea4e115e3ad
    Partial-Bug: 1834218
    Depends-On: https://review.opendev.org/#/c/703517/
 ...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.