Adding service-parameter to the kube_apiserver and apply refuses the connect to the apiserver

[Mihnea Saracin]

Brief Description
-----------------

Adding service-parameter to the kube_apiserver and apply refuses the connect the apiserver

Severity
--------
Major

Steps to Reproduce
------------------
1)Create the following service parameters and apply

system service-parameter-add kubernetes kube_apiserver oidc_client_id=stx-oidc-client-app
system service-parameter-add kubernetes kube_apiserver oidc_groups_claim=groups
system service-parameter-add kubernetes kube_apiserver oidc_issuer_url=https://[2620:10a:a001:a103::1230]:30556/dex
system service-parameter-add kubernetes kube_apiserver oidc_username_claim=email
system service-parameter-apply kubernetes

2)in the apiserver pod, it shows the following error

tail -f /var/log/pods/kube-system_kube-apiserver-controller-1_6bd59d46b5ec41f3b5a4a7c993ccf0b8/kube-apiserver/0.log
2021-11-08T14:33:25.389955265Z stderr F I1108 14:33:25.389766 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
2021-11-08T14:33:59.248643032Z stderr F I1108 14:33:59.248424 1 client.go:360] parsed scheme: "passthrough"
2021-11-08T14:33:59.248698316Z stderr F I1108 14:33:59.248505 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://[abcd:205::1]:2379 <nil> 0 <nil>}] <nil> <nil>}
2021-11-08T14:33:59.248722968Z stderr F I1108 14:33:59.248528 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
2021-11-08T14:34:40.636555426Z stderr F I1108 14:34:40.636284 1 client.go:360] parsed scheme: "passthrough"
2021-11-08T14:34:40.636589916Z stderr F I1108 14:34:40.636340 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://[abcd:205::1]:2379 <nil> 0 <nil>}] <nil> <nil>}
2021-11-08T14:34:40.636605817Z stderr F I1108 14:34:40.636349 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
2021-11-08T14:35:20.321415727Z stderr F I1108 14:35:20.321220 1 client.go:360] parsed scheme: "passthrough"
2021-11-08T14:35:20.321517025Z stderr F I1108 14:35:20.321283 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://[abcd:205::1]:2379 <nil> 0 <nil>}] <nil> <nil>}
2021-11-08T14:35:20.321554612Z stderr F I1108 14:35:20.321291 1 clientconn.go:948] ClientConn switching balancer to "pick_first"

2021-11-08T14:36:02.505849378Z stderr F I1108 14:36:02.505689 1 client.go:360] parsed scheme: "passthrough"
2021-11-08T14:36:02.505906896Z stderr F I1108 14:36:02.505729 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://[abcd:205::1]:2379 <nil> 0 <nil>}] <nil> <nil>}
2021-11-08T14:36:02.505930277Z stderr F I1108 14:36:02.505739 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
2021-11-08T14:36:18.304424409Z stderr F I1108 14:36:18.304245 1 controller.go:89] Shutting down OpenAPI AggregationController
2021-11-08T14:36:18.304461817Z stderr F I1108 14:36:18.304342 1 controller.go:123] Shutting down OpenAPI controller
2021-11-08T14:36:18.304469621Z stderr F I1108 14:36:18.304358 1 dynamic_cafile_content.go:182] Shutting down request-header::/etc/kubernetes/pki/front-proxy-ca.crt
2021-11-08T14:36:18.304477035Z stderr F I1108 14:36:18.304377 1 dynamic_serving_content.go:145] Shutting down aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key
2021-11-08T14:36:18.304482681Z stderr F I1108 14:36:18.304247 1 controller.go:181] Shutting down kubernetes service endpoint reconciler
2021-11-08T14:36:18.304491518Z stderr F I1108 14:36:18.304391 1 dynamic_cafile_content.go:182] Shutting down request-header::/etc/kubernetes/pki/front-proxy-ca.crt
2021-11-08T14:36:18.30451468Z stderr F I1108 14:36:18.304396 1 cluster_authentication_trust_controller.go:463] Shutting down cluster_authentication_trust_controller controller
2021-11-08T14:36:18.304546182Z stderr F I1108 14:36:18.304410 1 crdregistration_controller.go:142] Shutting down crd-autoregister controller
2021-11-08T14:36:18.304555323Z stderr F I1108 14:36:18.304418 1 establishing_controller.go:87] Shutting down EstablishingController
2021-11-08T14:36:18.304561606Z stderr F I1108 14:36:18.304422 1 autoregister_controller.go:165] Shutting down autoregister controller
2021-11-08T14:36:18.304565563Z stderr F I1108 14:36:18.304426 1 tlsconfig.go:255] Shutting down DynamicServingCertificateController
2021-11-08T14:36:18.304569546Z stderr F I1108 14:36:18.304433 1 apiapproval_controller.go:198] Shutting down KubernetesAPIApprovalPolicyConformantConditionController
2021-11-08T14:36:18.304576454Z stderr F I1108 14:36:18.304440 1 available_controller.go:487] Shutting down AvailableConditionController
2021-11-08T14:36:18.304592301Z stderr F I1108 14:36:18.304446 1 nonstructuralschema_controller.go:204] Shutting down NonStructuralSchemaConditionController
2021-11-08T14:36:18.304609583Z stderr F I1108 14:36:18.304448 1 dynamic_serving_content.go:145] Shutting down serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key
2021-11-08T14:36:18.304618375Z stderr F I1108 14:36:18.304463 1 apf_controller.go:303] Shutting down API Priority and Fairness config worker
2021-11-08T14:36:18.304622866Z stderr F I1108 14:36:18.304463 1 crd_finalizer.go:278] Shutting down CRDFinalizer
2021-11-08T14:36:18.30462885Z stderr F I1108 14:36:18.304400 1 dynamic_cafile_content.go:182] Shutting down client-ca-bundle::/etc/kubernetes/pki/ca.crt
2021-11-08T14:36:18.304633839Z stderr F I1108 14:36:18.304479 1 naming_controller.go:302] Shutting down NamingConditionController
2021-11-08T14:36:18.304639095Z stderr F I1108 14:36:18.304386 1 customresource_discovery_controller.go:245] Shutting down DiscoveryController
2021-11-08T14:36:18.304649489Z stderr F I1108 14:36:18.304380 1 apiservice_controller.go:131] Shutting down APIServiceRegistrationController
2021-11-08T14:36:18.304655324Z stderr F I1108 14:36:18.304497 1 dynamic_cafile_content.go:182] Shutting down client-ca-bundle::/etc/kubernetes/pki/ca.crt
2021-11-08T14:36:18.305300873Z stderr F I1108 14:36:18.305213 1 secure_serving.go:241] Stopped listening on [::]:6443
2021-11-08T14:36:18.305802129Z stderr F E1108 14:36:18.305732 1 controller.go:184] Get "https://[::1]:6443/api/v1/namespaces/default/endpoints/kubernetes": dial tcp [::1]:6443: connect: connection refused
2021-11-08T14:36:37.566185524Z stderr F I1108 14:36:37.565988 1 client.go:360] parsed scheme: "passthrough"
2021-11-08T14:36:37.566243661Z stderr F I1108 14:36:37.566035 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://[abcd:205::1]:2379 <nil> 0 <nil>}] <nil> <nil>}
2021-11-08T14:36:37.566269606Z stderr F I1108 14:36:37.566045 1 clientconn.go:948] ClientConn switching balancer to "pick_first"

Expected Behavior
------------------

service parameters to the kubeapi server should be applied without any errors

Actual Behavior
----------------

shows "connect: connection refused"

Reproducibility

100%

System Configuration
--------------------

All

Branch/Pull Time/Commit
-----------------------

stx master built on BUILD_ID="2021-10-25_00-00-10"

Test Activity
-------------

regression analysis

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/820418

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/820419

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/820418
Committed: https://opendev.org/starlingx/stx-puppet/commit/a1b99570ae0fd387cb0fe975d8974dcb4edbf367
Submitter: "Zuul (22348)"
Branch: master

commit a1b99570ae0fd387cb0fe975d8974dcb4edbf367
Author: Mihnea Saracin <email address hidden>
Date: Sat Dec 4 17:12:27 2021 +0200

    Wait for kube apiserver after apply service parameter

    When doing a: "system service-parameter-apply kubernetes"
    the kube apiserver will be restarted but the service
    parameter command will return regardless of the apiserver
    state. And if we run some 'kubectl' commands after that, they will
    fail until the apiserver is back online.
    To avoid these bad scenarios, we will wait for the kube apiserver
    to be up after we apply the service parameter.

    Closes-Bug: 1953183
    Change-Id: Ic3cc3728a8eb363a06f268c904680f30c40b5d44
    Signed-off-by: Mihnea Saracin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Mihnea Saracin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/config/+/820419
Reason: No longer needed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/821395

[Ghada Khalil (gkhalil)]

Re-opening; the original fix was reverted as it introduced another issue as reported in https://bugs.launchpad.net/starlingx/+bug/1954333

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/821473

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/821473
Committed: https://opendev.org/starlingx/stx-puppet/commit/61b72946b8eb81a34c7559b4a0015a3af23f179e
Submitter: "Zuul (22348)"
Branch: master

commit 61b72946b8eb81a34c7559b4a0015a3af23f179e
Author: David Sullivan <email address hidden>
Date: Fri Dec 10 14:27:28 2021 -0600

    Correct ordering when upgrading etcd certs

    During an upgrade activate we generate new etcd certs. As part of this
    process we need to restart etcd and direct the kube-apiserver to use the
    new certs.

    With this update we'll restart the etcd server before updating the
    kube-apiserver. Finally we'll wait for the kube-apiserver to be ready.
    Waiting for the apiserver is necessary so subsequent sysinv operations
    can interact with kubeadm.
    https://bugs.launchpad.net/starlingx/+bug/1954488 and
    https://bugs.launchpad.net/starlingx/+bug/1953183 are seen if we do not
    wait for the apiserver.

    Note curl -f is used to ensure the readyz call returns 200/OK.

    Closes-Bug: 1954488
    Closes-Bug: 1953183
    Change-Id: I3b31153d8779262c0bf5e57526b41421689bad36
    Signed-off-by: David Sullivan <email address hidden>

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Mihnea Saracin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/821395
Reason: Another commit fixed the bug: https://review.opendev.org/c/starlingx/stx-puppet/+/821473

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.6.0
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/821589

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/821589
Committed: https://opendev.org/starlingx/stx-puppet/commit/15aa007c2510f1620cee798527096ac57aae1188
Submitter: "Zuul (22348)"
Branch: r/stx.6.0

commit 15aa007c2510f1620cee798527096ac57aae1188
Author: David Sullivan <email address hidden>
Date: Fri Dec 10 14:27:28 2021 -0600

    Correct ordering when upgrading etcd certs

    During an upgrade activate we generate new etcd certs. As part of this
    process we need to restart etcd and direct the kube-apiserver to use the
    new certs.

    With this update we'll restart the etcd server before updating the
    kube-apiserver. Finally we'll wait for the kube-apiserver to be ready.
    Waiting for the apiserver is necessary so subsequent sysinv operations
    can interact with kubeadm.
    https://bugs.launchpad.net/starlingx/+bug/1954488 and
    https://bugs.launchpad.net/starlingx/+bug/1953183 are seen if we do not
    wait for the apiserver.

    Note curl -f is used to ensure the readyz call returns 200/OK.

    Closes-Bug: 1954488
    Closes-Bug: 1953183
    Change-Id: I3b31153d8779262c0bf5e57526b41421689bad36
    Signed-off-by: David Sullivan <email address hidden>
    (cherry picked from commit 61b72946b8eb81a34c7559b4a0015a3af23f179e)