Fail to unlock controller after Backup and Restore when oidc is configured

[Mihnea Saracin]

Brief Description
-----------------
Controller-0 failed to unlock after Backup and Restore a system with oidc-auth-apps installed.

Severity
--------
Major

Steps to Reproduce
------------------
- Install auth-oidc-apps
- Backup
- Restore
- Unlock controller-0

Expected Behavior
------------------
Successful unlock

Actual Behavior
----------------
Failed unlock

Reproducibility
---------------
100%

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2020-03-22 23:00:00 +0000"

Timestamp/Logs
--------------

  [sysadmin@controller-0 ~(keystone_admin)]$ system application-list
  +---------------------+---------+-------------------------------+---------------+----------+-----------+
  | application | version | manifest name | manifest file | status | progress |
  +---------------------+---------+-------------------------------+---------------+----------+-----------+
  | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | applied | completed |
  | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed |
  +---------------------+---------+-------------------------------+---------------+----------+-----------+

  [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock 1
  'NoneType' object is not iterable
  Traceback (most recent call last):

    File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data
      **args)

    File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
      result = getattr(proxyobj, method)(ctxt, **kwargs)

    File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1687, in configure_ihost
      self._configure_controller_host(context, host)

    File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1365, in _configure_controller_host
      context, config_uuid, config_dict, force=True)

    File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 8460, in _config_apply_runtime_manifest
      self.evaluate_app_reapply(context, app_name)

    File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10777, in evaluate_app_reapply
      app.sync_overrides_dir, app.charts, app.name, None)

  TypeError: 'NoneType' object is not iterable

Test Activity
-------------
Feature Testing

Workaround
----------
Set oidc-auth-apps application status to 'uploaded'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715495

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - B&R failing for a specific config; should be addressed for stx.4.0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715495
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=bf103f3c54eb45c26d52a43c35339d1d863a42de
Submitter: Zuul
Branch: master

commit bf103f3c54eb45c26d52a43c35339d1d863a42de
Author: Mihnea Saracin <email address hidden>
Date: Fri Mar 27 18:19:02 2020 +0200

    Fix B&R when the controller needs to be unlocked

    After running the restore playbook, all the applications
    should be in an uploaded state. But they are in an
    applied state instead, making the controller-0
    unable to unlock.

    Closes-Bug: 1869403
    Change-Id: I8bd9c51e250969cc334d52b78c616f9ad082afd8
    Signed-off-by: Mihnea Saracin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716133

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716133
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=ddcb11f4b773f4b3190663defe3ba0f3ec4201c8
Submitter: Zuul
Branch: f/centos8

commit bf103f3c54eb45c26d52a43c35339d1d863a42de
Author: Mihnea Saracin <email address hidden>
Date: Fri Mar 27 18:19:02 2020 +0200

    Fix B&R when the controller needs to be unlocked

    After running the restore playbook, all the applications
    should be in an uploaded state. But they are in an
    applied state instead, making the controller-0
    unable to unlock.

    Closes-Bug: 1869403
    Change-Id: I8bd9c51e250969cc334d52b78c616f9ad082afd8
    Signed-off-by: Mihnea Saracin <email address hidden>

commit 6e875971afeaf1378c2c8aeb845359459838ce30
Author: Stefan Dinescu <email address hidden>
Date: Sat Mar 21 16:57:57 2020 +0200

    Fix Netapp port conflict

    By default, the Trident Netapp service opens port 8443 for
    HTTPS REST api usage. This conflicts with the port the
    Horizon dashboard uses on an HTTPS enabled setup (the port
    is also 8443).

    In order to fix this, we change the default port from 8443
    to 8678, but also make it configurable through ansible
    overrides.

    The Trident service also opens port 8001 for metrics usage.
    While that doesn't currently conflict with any other service
    on the system, I also made that configurable through
    ansible overrides, in case such a conflict appears in the
    future.

    Change-Id: I08db939acac6082f82b9e12e932d8289c7cecdeb
    Closes-bug: 1868382
    Signed-off-by: Stefan Dinescu <email address hidden>

commit 5a9ba6786e393f2cd93bfae8c3a8f09f0cf9eb26
Author: Robert Church <email address hidden>
Date: Thu Mar 19 19:08:17 2020 -0400

    Upversion Multus to 3.4

    Updates the Multus configuration to align with version 3.4

    Change-Id: Ifc236ccbbe4e559987d7ef522902f638062348ca
    Depends-On: https://review.opendev.org/#/c/714024/
    Story: 2006999
    Task: 39110
    Signed-off-by: Robert Church <email address hidden>

commit 6a261463f9ac0f81d9c7f054dd3cb10a51934d4a
Author: Robert Church <email address hidden>
Date: Wed Mar 18 22:01:03 2020 -0400

    Upversion Calico from 3.6 to 3.12

    Updates the Calico configuration to align with version 3.12. This
    introduces support for a Flex Volume Driver which requires enabling the
    --volume-plugin-dir option for kubelet, the --flex-volume-plugin-dir
    option for kube-controller-manager, and pulling the pod2daemon-flexvol
    image used by calico-node pods.

    Change-Id: I74bc5c53ffcb16c8e3c06cebf20eac296b9ccc65
    Story: 2006999
    Task: 39109
    Depends-On: https://review.opendev.org/#/c/714023
    Signed-off-by: Robert Church <email address hidden>

commit b35387f8bc40714e9633e6191267284b8af8ccee
Author: Stefan Dinescu <email address hidden>
Date: Thu Mar 19 18:13:26 2020 +0200

    Netapp: Fix handling of IPv6 addresses

    Using bash process subtitution to pass the file parameter
    to the "create backend" command doesn't work as the bash
    variable expansion...