WAD user cannot access the K8S API after applying Oidc app

[Andy]

Brief Description
-----------------
WAD user cannot access the K8S API after applying Oidc app

Severity
--------
Major: WAD user cannot access k8s API by oidc.

Steps to Reproduce
------------------
Following procedure to enable oidc:
- apply oidc service parameters
- generate local-dex.tls, dex-client-secret secrets
- helm overrides update for oidc-auth-apps
- apply oidc-auth-apps
- get oidc token by oidc-auth cli
- generate kubeconfig to be used by kubectl command
- run "sudo kubectl --kubeconfig /home/sysadmin/kubeconfig get pods"

Expected Behavior
------------------
WAD user should access the K8s api without any issues. The command lists the pods in the system.

Actual Behavior
----------------
[sysadmin@controller-0 ~(keystone_admin)]$ sudo kubectl --kubeconfig /home/sysadmin/kubeconfig get pods
error: You must be logged in to the server (Unauthorized)

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
STX master latest.

Last Pass
---------
Unknown

Timestamp/Logs
--------------

Following error is shown continuously on oidc dex pod:

[sysadmin@controller-0 ~(keystone_admin)]$ tail -f /var/log/pods/kube-system_oidc-dex-9ff9d96bd-m8pl8_cddae727-b9d7-4872-9406-a53c38dd902e/main/0.log

2022-03-04T15:23:05.05916354Z stderr F 2022/03/04 15:23:05 http: TLS handshake error from 128.224.48.231:36027: remote error: tls: bad certificate
2022-03-04T15:23:15.058167765Z stderr F 2022/03/04 15:23:15 http: TLS handshake error from 128.224.48.231:9743: remote error: tls: bad certificate
2022-03-04T15:23:25.059066316Z stderr F 2022/03/04 15:23:25 http: TLS handshake error from 128.224.48.231:24445: remote error: tls: bad certificate
2022-03-04T15:23:35.05875156Z stderr F 2022/03/04 15:23:35 http: TLS handshake error from 128.224.48.231:29302: remote error: tls: bad certificate
2022-03-04T15:23:45.05876631Z stderr F 2022/03/04 15:23:45 http: TLS handshake error from 128.224.48.231:31992: remote error: tls: bad certificate
2022-03-04T15:23:55.058705721Z stderr F 2022/03/04 15:23:55 http: TLS handshake error from 128.224.48.231:5552: remote error: tls: bad certificate

kube-apiserver pod has this error in logs:

E0615 05:53:22.726624 1 oidc.go:224] oidc authenticator: initializing plugin: oidc: issuer did not match the issuer returned by provider, expected "https://192.168.206.2:30556/dex" got "https://10.10.10.83:30556/dex"
E0615 05:53:32.725379 1 oidc.go:224] oidc authenticator: initializing plugin: oidc: issuer did not match the issuer returned by provider, expected "https://192.168.206.2:30556/dex" got "https://10.10.10.83:30556/dex"
E0615 05:53:33.586420 1 authentication.go:63] "Unable to authenticate the request" err="invalid bearer token"

Test Activity
-------------
Regression Testing

Workaround
----------
Update /etc/kubernetes/manifests/kube-apiserver.yaml,
set oidc-issuer-url pointing to OAM floating IP:

    - --oidc-issuer-url=https://<OAM floating IP>:30556/dex

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/846237

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/846237
Committed: https://opendev.org/starlingx/stx-puppet/commit/f6a29166ec00bd1a94459d838fa3f9f7117bf6f0
Submitter: "Zuul (22348)"
Branch: master

commit f6a29166ec00bd1a94459d838fa3f9f7117bf6f0
Author: Andy Ning <email address hidden>
Date: Thu Jun 16 15:52:49 2022 -0400

    Fix WAD user cannot access k8s API by oidc

    Currently when oidc-auth-apps is applied and oidc service
    parameters are applied, kube-apiserver's oidc_issuer_url points
    to cluster host floating IP instead of the OAM floating IP. This
    causes mis-match of oidc issuer that kube-apiserver is configured
    and the actual oidc issuer's IP address. User can no longer access
    k8s API even with a valid token.

    The issue is introduced by a sed substitution in
    kube-apiserver-change-params.erb where it replaces all the OAM IPs
    with kube-apisever's advertise address, including oidc-issuer-url.
    This fixed it by excluding oidc-issuer-url from the substitution.

    Test Plan for CentOS and Debian:
    PASS: oidc service parameters apply, helm overrides update and
          oidc-auth-apps apply
    PASS: run oidc-auth cli to get a token
    PASS: use the token to access k8s API by kubectl

    Closes-Bug: 1971500
    Closes-Bug: 1979006
    Signed-off-by: Andy Ning <email address hidden>
    Change-Id: I19d434c6322b4423d2e5b1732ff8af3f486b73f2