Cant access dex client when all the dex pods running on standby controller

[ayyappa]

Brief Description
-----------------
Dex client url is not accessible when all the dex pods running on standby controller

Severity
--------
Major

Steps to Reproduce
------------------
1)oidc-dex, stx-oidc-client is created on both the controllers initially
2)On the active controller-0, lock/unlock the standby controller
3)Now all the 4 pods move to controller-0
4)Now swact the controller-0
5)On the active controller-1, all pods still on standby controller-0
[sysadmin@controller-1 ~(keystone_admin)]$ kubectl get pods -n kube-system -o wide | grep oidc
oidc-dex-6b56c574bd-5wrcd 1/1 Running 0 30m dead:beef::8e22:765f:6121:eb5c controller-0 <none> <none>
oidc-dex-6b56c574bd-f2vmp 1/1 Running 1 69m dead:beef::8e22:765f:6121:eb52 controller-0 <none> <none>
stx-oidc-client-69f5785dd7-gsplr 1/1 Running 0 69m dead:beef::8e22:765f:6121:eb53 controller-0 <none> <none>
stx-oidc-client-69f5785dd7-ktwtp 1/1 Running 1 30m dead:beef::8e22:765f:6121:eb5b controller-0 <none> <none>

6)Try the client url https://<ip>:30555 which fails since there is no oidc-dex,stx-oidc-client running on active controller.

Expected Behavior
------------------
After lock/unlock the standby, the pods should be assigned back to the standby controller

Actual Behavior
----------------
client url doesn't work when all the pods are running on standby

Reproducibility
---------------
100%

System Configuration
--------------------
tested on all the following systems
duplex ipv6 r430_3_4

Branch/Pull Time/Commit
-----------------------
2020-03-02

Last Pass
---------
This is a new test scenario

Timestamp/Logs
--------------
2020-03-02T19:35:09.405194168Z

Test Activity
-------------
Feature Testing

Workaround
----------
Delete the pods manually, which creates the pods on active controller automatically solves the issue
[sysadmin@controller-1 ~(keystone_admin)]$ kubectl get pods -n kube-system -o wide | grep oidc
oidc-dex-6b56c574bd-bn6nw 1/1 Running 0 2m57s dead:beef::8e22:765f:6121:eb64 controller-0 <none> <none>
oidc-dex-6b56c574bd-r7vzn 1/1 Running 0 3m9s dead:beef::a4ce:fec1:5423:e306 controller-1 <none> <none>
stx-oidc-client-69f5785dd7-7h2rl 1/1 Running 3 2m39s dead:beef::a4ce:fec1:5423:e307 controller-1 <none> <none>
stx-oidc-client-69f5785dd7-ktwtp 1/1 Running 1 37m dead:beef::8e22:765f:6121:eb5b controller-0 <none> <none>

[Ghada Khalil (gkhalil)]

stx.4.0 / high priority - issue w/ active directory feature which is an stx.4.0 deliverable

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713945
Committed: https://git.openstack.org/cgit/starlingx/oidc-auth-armada-app/commit/?id=dbef0350666cd85ed54e245b08308e8039414fc3
Submitter: Zuul
Branch: master

commit dbef0350666cd85ed54e245b08308e8039414fc3
Author: Teresa Ho <email address hidden>
Date: Thu Mar 19 13:17:38 2020 -0400

    Add affinity to dex and oidc-client helm charts

    When the standby controller is locked, the oidc pods are scheduled onto
    the active controller. When the standby controller is unlocked, the oidc
    pods are not scheduled on the standby. This results in four oidc pods on
    the active controller.
    This update sets the pod anti-affinity rule such that the pod does
    not get scheduled onto a node if the node already has the same pod
    running.

    Closes-Bug: 1865565

    Change-Id: I19113ab5a11f0691a0cdba9657138cc3251788cc
    Signed-off-by: Teresa Ho <email address hidden>

[ayyappa (mantri425)]

Fix working fine in the BUILD_ID="2020-03-25_21-02-05"