Helm override update "oidc-auth-apps" with new values doesn't take affect

[ayyappa]

Brief Description
-----------------
Override the oidc-auth-apps with new values and apply without removing it, the pods still runs on old values and new values doesn't take affect

Severity
--------
Major

Steps to Reproduce
------------------
1)After controller is unlocked, create kubernetes secret for running dex and oidc-client:
kubectl create secret tls local-dex.tls --cert=ssl/dex-cert.pem --key=ssl/dex-key.pem -n kube-system
kubectl create secret generic dex-client-secret --from-file=/home/sysadmin/ssl/dex-ca.pem -n kube-system

2)copy the dex-overrride.yaml file
cat dex-overrides.yaml
config:
  expiry:
    idTokens: "20m"
  connectors:
  - type: ldap
    name: OpenLDAP
    id: ldap
    config:
      host: pv-windows-acti.cumulus.wrs.com:636
      rootCA: /etc/ssl/certs/adcert/AD_CA.cer
      insecureNoSSL: false
      insecureSkipVerify: false
      bindDN: cn=Administrator,cn=Users,dc=cumulus,dc=wrs,dc=com
      bindPW: Li69nux*
      usernamePrompt: Username
      userSearch:
        baseDN: ou=Users,ou=Titanium,dc=cumulus,dc=wrs,dc=com
        filter: "(objectClass=user)"
        username: sAMAccountName
        idAttr: sAMAccountName
        emailAttr: sAMAccountName
        nameAttr: displayName
extraVolumes:
- name: certdir
  secret:
    secretName: wadcert
extraVolumeMounts:
- name: certdir
  mountPath: /etc/ssl/certs/adcert

4)and apply the application:

system helm-override-update oidc-auth-apps dex kube-system --values /home/sysadmin/dex-overrides.yaml
system application-apply oidc-auth-apps

[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+---------+-------------------------------+---------------+---------+-----------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+---------+-------------------------------+---------------+---------+-----------+
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | applied | completed |
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |

5)Now change the values of the dex-override.yaml file like "id expiry" and override,apply as follows
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update oidc-auth-apps dex kube-system --values /home/sysadmin/ssl/dex-overrides.yaml
+----------------+------------------------------------------------------------------+
| Property | Value |
+----------------+------------------------------------------------------------------+
| name | dex |
| namespace | kube-system |
| user_overrides | config: |
| | connectors: |
| | - config: |
| | bindDN: cn=Administrator,cn=Users,dc=cumulus,dc=wrs,dc=com |
| | bindPW: Li69nux* |
| | host: pv-windows-acti.cumulus.wrs.com:636 |
| | insecureNoSSL: false |
| | insecureSkipVerify: false |
| | rootCA: /etc/ssl/certs/adcert/AD_CA.cer |
| | userSearch: |
| | baseDN: ou=Users,ou=Titanium,dc=cumulus,dc=wrs,dc=com |
| | emailAttr: sAMAccountName |
| | filter: (objectClass=user) |
| | idAttr: sAMAccountName |
| | nameAttr: displayName |
| | username: sAMAccountName |
| | usernamePrompt: Username |
| | id: ldap |
| | name: OpenLDAP |
| | type: ldap |
| | expiry: |
| | idTokens: 60s |
| | extraVolumeMounts: |
| | - mountPath: /etc/ssl/certs/adcert |
| | name: certdir |
| | extraVolumes: |
| | - name: certdir |
| | secret: |
| | secretName: wadcert |
| | |
+----------------+------------------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-apply oidc-auth-apps
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | True |
| app_version | 1.0-0 |
| created_at | 2020-02-24T15:24:35.171634+00:00 |
| manifest_file | manifest.yaml |
| manifest_name | oidc-auth-manifest |
| name | oidc-auth-apps |
| progress | None |
| status | applying |
| updated_at | 2020-02-24T15:53:52.303171+00:00 |
+---------------+----------------------------------+

7)Try to retrieve the token, the token still returns with 20m expiry instead of 60s

Expected Behavior
------------------
The new values should take affect automatically without remove/ reapply

Actual Behavior
----------------
Failed to update the pods with new values

Reproducibility
---------------
100%

System Configuration
--------------------
tested on all the following systems
simple wcp_122 ipv4

Branch/Pull Time/Commit
-----------------------
2020-02-22

Last Pass
---------
This is a new test scenario

Timestamp/Logs
--------------
2020-02-24 15:02:21.182

Test Activity
-------------
Feature Testing

Workaround
----------
Delete dex pod , which the pod is created automatically with new values.

[ayyappa (mantri425)]

Logs attached

[Jerry Sun (jerry-sun-u)]

can also work around the issue by removing the application, and then applying the application

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - related to the Active Directory 4.0 feature, but there is a workaround.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728553

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728553
Committed: https://git.openstack.org/cgit/starlingx/oidc-auth-armada-app/commit/?id=663765f109a6c70468da67da0c467645a4098f13
Submitter: Zuul
Branch: master

commit 663765f109a6c70468da67da0c467645a4098f13
Author: Jerry Sun <email address hidden>
Date: Fri May 15 15:03:51 2020 -0400

    Automatically roll deployments when config is changed

    Automatically roll deployments when config is changed. Otherwise, the
    new config is pushed but the behavior is not. We cannot do the exact
    thing upstream did because of an armada bug. We are doing what
    monitor-armada-app does when it runs into the same issue.

    Closes-Bug: 1864504

    Change-Id: Ib714694c93b57a28d37da595ba2efd6885bf9307
    Signed-off-by: Jerry Sun <email address hidden>

[ayyappa (mantri425)]

Fix verified on ip_18_19 standard system with buid id 2020-05-20_21-00-00