DC: subcloud platform-integ-apps reapply failed at retrieving central registry credential

[Yang Liu]

Brief Description
-----------------
On DC subclouds, platform-integ-apps fails on any reapply.
It fails at retrieving barbecan secret, due to the original keystone user that was used to create the barbecan secret - was created before DC sync and was then replaced by the new admin user synced down from central.

Severity
--------
Critical

Steps to Reproduce
------------------
- Install and configure DC
- Do something that triggers reapply of platform-integ-apps, such as lock/unlock host on an AIO subcloud, or manually re-apply platform-integ-apps on subclouds

Expected Behavior
------------------
- platform-integ-apps reapplied successfully

Actual Behavior
----------------
- platform-integ-apps apply-failed with error: Unable to get the credentials to access registry registry.central:

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Distributed Cloud system

Branch/Pull Time/Commit
-----------------------
20191102

Last Pass
---------
None
This issue was uncovered when subcloud started to pull images from registry.central. However, this issue would have been seen with any private registry even without the registry.central change.

Timestamp/Logs
--------------
Analysis from Angie Wang:
Barbican secrets are created by admin user under admin project at subcloud bootstrap time. Subcloud admin user/project is created by subcloud keystone during bootstrap. After system is unlocked, dbdcsync is up, it syncs keystone admin user/project between system controller and subcloud, so the subcloud admin user/project ‘s id s become same as the ids of the system controller. The original admin user/project of subcloud get deleted. When applying application, it will fail to find the barbican secret as it’s created by/under the original subcloud admin.

The user/project has not been synced during the initial platform-form-integ apply, so it passed at the first time.

# System controller
[sysadmin@controller-0 ~(keystone_admin)]$ openstack user list
+----------------------------------+-----------+
| ID | Name |
+----------------------------------+-----------+
| 08c71a9bc01d426abd0239213c7fa215 | mtce |
| a62f1b82df404537931ce7ff9859ab75 | fm |
| c143937dfee94f809b79aa92cd7d0ec8 | admin |
| 20c0b85e0fcf4d409d1d28a63f53501a | barbican |
| 71a55e59fb6042a4846e9bd742524bb6 | sysinv |
| 5487a452d6764e0f9fbbd734015d580f | patching |
| 39818c1597454c4485e229c86b919b75 | vim |
| 818dc89581ad40c2b9f4617007db6ae7 | dcmanager |
| d81cea56313f4386b2e71d436190b31b | dcorch |
| 97364a317b64471f937ecd7e20996960 | dcdbsync |
| 2c160aae50ef42b7a4cdfb611ee1147e | smapi |
+----------------------------------+-----------+

[sysadmin@controller-0 ~(keystone_admin)]$ openstack project list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 5f933f67321c428e94ce2675669f22ac | admin |
| 805d22cd9c884998b5fad16c4bc44e11 | services |
+----------------------------------+----------+

# Subcloud 6
Secrets were created by admin user with id “de0cab3b648e4836a974ba315adcca13” and project id “bf3037a9-ad0e-4b8f-bb45-5b89b0272695”

barbican=# select * from secrets where id='4cf054c3-3d54-48d6-9d98-c4523a7bd9ad';
                  id | created_at | updated_at | deleted_at | deleted | status | name | expiration | algorithm | bit_length | mode | secret_type | creator_id |
        project_id
--------------------------------------+----------------------------+----------------------------+------------+---------+--------+---------------------+------------+-----------+------------+------+-------------+----------------------------------+------
--------------------------------
4cf054c3-3d54-48d6-9d98-c4523a7bd9ad | 2019-11-03 00:09:05.993246 | 2019-11-03 00:09:05.996436 | | f | ACTIVE | k8s-registry-secret | | aes | 256 | cbc | opaque | de0cab3b648e4836a974ba315adcca13 | bf303
7a9-ad0e-4b8f-bb45-5b89b0272695

(1 row)

After dbdcsync, Ids changed on subcloud.
[sysadmin@controller-0 ~(keystone_admin)]$ openstack user list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| f92e321bda7f4954860aba481bf91c47 | fm |
| f0699ea2db964445bd55d1e247cca67e | dcdbsync |
| 018621060fbb45269f49ea57d500fc59 | smapi |
| c143937dfee94f809b79aa92cd7d0ec8 | admin |
| 08c71a9bc01d426abd0239213c7fa215 | mtce |
| 20c0b85e0fcf4d409d1d28a63f53501a | barbican |
| 71a55e59fb6042a4846e9bd742524bb6 | sysinv |
| 5487a452d6764e0f9fbbd734015d580f | patching |
| 39818c1597454c4485e229c86b919b75 | vim |
+----------------------------------+----------+

[sysadmin@controller-0 ~(keystone_admin)]$ openstack project list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 5f933f67321c428e94ce2675669f22ac | admin |
| 805d22cd9c884998b5fad16c4bc44e11 | services |
+----------------------------------+----------+

Secret get will fail
[sysadmin@controller-0 ~(keystone_admin)]$ openstack secret get 4cf054c3-3d54-48d6-9d98-c4523a7bd9ad -p
4xx Client error: Forbidden: Secret payload retrieval attempt not allowed - please review your user/project privileges
Forbidden: Secret payload retrieval attempt not allowed - please review your user/project privileges

Test Activity
-------------
 Regression Testing

[Ghada Khalil (gkhalil)]

Marking as stx.3.0 / high priority - issue w/ Distributed Cloud

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693542

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693542
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=5afd5f90b29f6e097824f7c6f2fe7762597d9ad6
Submitter: Zuul
Branch: master

commit 5afd5f90b29f6e097824f7c6f2fe7762597d9ad6
Author: Andy Ning <email address hidden>
Date: Tue Nov 5 00:12:26 2019 -0500

    update Barbican admin secret's user/project IDs during bootstrap

    In a DC system when subcloud is managed, keystone user/project IDs are
    synced with Central Cloud, including admin user and project. But the
    admin's secrets in Barbian still use the original user/project IDs,
    causing docker registry access failure when platform-integ-apps is
    reapplied.

    This change added a patch to keystone puppet manifest, that updates
    keystone admin user/project IDs to be the same as Central Cloud right
    after keystone is bootstrapped during subcloud deployment. This way any
    referece to admin user/project IDs after bootstrap will be using the
    IDs same as Central Cloud, including the ones in Barbican. This will
    solve the problem of retrieving central registry credential failure
    when platform-integ-apps is reapplied.

    Change-Id: I509a06b4b810620a1b3648837726f7f2771162a5
    Closes-Bug: 1851247
    Signed-off-by: Andy Ning <email address hidden>

[Ghada Khalil (gkhalil)]

Re-opening as two additional reviews still need to merge:
https://review.opendev.org/#/c/693578/
https://review.opendev.org/#/c/693579/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693578
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=c0be9e8f7981419e6682726b4c36122b55438974
Submitter: Zuul
Branch: master

commit c0be9e8f7981419e6682726b4c36122b55438974
Author: Angie Wang <email address hidden>
Date: Fri Nov 8 13:59:01 2019 -0500

    Pass keystone admin user/project IDs to subcloud

    This commit updates to query system controller keystone admin
    user/project IDs and write them into subcloud bootstrap overrides
    file. These data will be used at subcloud bootstrap time to update
    subcloud keystone admin user/project IDs to be same as the system
    controller.

    Change-Id: Iddbe51e87b3f3876b88bbdeef4b0b05e46443612
    Partial-Bug: 1851247
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693579
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=809afe783f739551fbee3be1cf72d0023ebeccc7
Submitter: Zuul
Branch: master

commit 809afe783f739551fbee3be1cf72d0023ebeccc7
Author: Angie Wang <email address hidden>
Date: Fri Nov 8 13:27:23 2019 -0500

    Write keystone admin user and project IDs to subcloud puppet hieradata

    This commit adds a task to write system controller keystone admin
    user/projects IDs to puppet hieradata at subcloud bootstrap time.
    The hieradata will be used during keystone bootstrap to update subcloud
    keystone admin and project IDs to be same as the system controller's
    admin user and project IDs.

    Change-Id: Icdb3a5fde3901b3e5fd3ae3ada03f784b8c33f1a
    Closes-Bug: 1851247
    Depends-On: https://review.opendev.org/#/c/693578/
    Related-To: https://review.opendev.org/#/c/693542/
    Signed-off-by: Angie Wang <email address hidden>

[Yang Liu (yliu12)]

This is verified on 10-06 load.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/698561

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698561
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=9035cd1be8aa3138691c6c99219030dfbe77ebaf
Submitter: Zuul
Branch: f/centos8

commit 4aa661ce5666220d6beb2a3a3fac987cba4feb74
Author: Martin, Chen <email address hidden>
Date: Thu Nov 21 10:28:13 2019 +0800

    Build layering
    Rebase tarball for i40e Driver
    Rebase srpm for systemd 219-67.el7
    Rebase srpm for sudo
    Rebase srpm for ntp

    Depends-On: https://review.opendev.org/#/c/695061/
    Depends-On: https://review.opendev.org/#/c/695560/
    Depends-On: https://review.opendev.org/#/c/695637/
    Depends-On: https://review.opendev.org/#/c/695983/

    Story: 2006166
    Task: 37570

    Change-Id: I7f33e0fb1319df3421318c4927d2a5675a490273
    Signed-off-by: Martin, Chen <email address hidden>

commit 5d854355d873702b78ff6aa8c6fddc025c45be2d
Author: Jim Somerville <email address hidden>
Date: Mon Nov 25 16:07:17 2019 -0500

    Uprev ntp to version 4.2.6p5-29.el7

    This solves:
    ntp: Stack-based buffer overflow in ntpq and ntpdc allows
    denial of service or code execution (CVE-2018-12327)

    See the announcement link:

    https://lists.centos.org/pipermail/centos-cr-announce/2019-August/006016.html

    for more details.

    Here we refresh the meta patches and correct the crime of
    "name of patch file differs from git format-patch". We
    also clean up the commit short logs.

    Change-Id: I263465d85f06096296fdd478a302eb110ab1259c
    Closes-Bug: 1849197
    Depends-On: https://review.opendev.org/#/c/695983
    Signed-off-by: Jim Somerville <email address hidden>

commit 11fd5d9cd48a1539b9c7a4ebc8aaad69ed24ae5b
Author: Dan Voiculeasa <email address hidden>
Date: Thu Nov 21 15:01:36 2019 +0200

    ceph-init-wrapper: Detect stuck peering OSDs and restart them

    OSDs might become stuck peering.
    Recover from such state.

    Closes-bug: 1851287

    Change-Id: I2ef1a0e93d38c3d041ee0c5c1e66a4ac42785a68
    Signed-off-by: Dan Voiculeasa <email address hidden>

commit f30cb74fef4b97721010ca9bc6a6b6dde03c4add
Author: Robin Lu <email address hidden>
Date: Fri Nov 22 11:01:27 2019 +0800

    Update sudo srpm patch for CVE bug

    To fix below CVE, we will use sudo-1.8.23-4.el7_7.1.src.rpm
    And we have to update some patches according to new srpm.
    https://lists.centos.org/pipermail/centos-announce/2019-October/023499.html

    CVE bug: CVE-2019-14287: sudo: can bypass certain policy blacklists

    Closes-Bug: 1852825
    Depends-On: https://review.opendev.org/#/c/695637/
    Change-Id: Ifc0a3423464fafce06cd504d9b427fc3433fb756
    Signed-off-by: Robin Lu <email address hidden>

commit 0231aba5cdcb96b15106591acfff280159050366
Author: Jim Somerville <email address hidden>
Date: Thu Nov 21 15:54:15 2019 -0500

    Uprev systemd to version 219-67.el7

    This solves:
    systemd: line splitting via fgets() allows for state injection
    during daemon-reexec (CVE-2018-15686)

    along with some other less critical issues. See the security
    announcement link:

   ...