DistributedCloud: armada 'service' project error on initial subcloud sync

[John Kung]

Brief Description
-----------------
In a Distributed Cloud System, on initial manage of subcloud, the initial sync enounters an error attempting to assign role to a 'service' project. The identity sync alarm requires an extra audit (20 minutes) cycle to clear due to error on missing 'service' project.

The 'service' project was introduced for Helm v3 and Armada.

Severity
--------
Minor: System/Feature is usable with minor issue

Steps to Reproduce
------------------
In a Distributed Cloud system, add a subcloud and perform the initial manage.

Expected Behavior
------------------
Initial sync should complete without error.

Actual Behavior
----------------
The identity sync alarm requires an audit cycle to clear; the following ERROR log is noted on the initial manage:

2020-12-01 08:03:09.113 106271 ERROR dcorch.engine.sync_services.identity [-] subcoud1/identity: Unable to assign role to user on project reference Resource(id=29,master_id='466f984c13d648c78f18a9ec8d4beb99_5719d84bc8924a3c897bc22056278cde_cbb11e52b38d4f7ab8ca89694bbc3406',resource_type='project_role_assignments',uuid=4fd70497-c7da-4f83-ac0b-fc31a1555fe3):466f984c13d648c78f18a9ec8d4beb99, cannot find equivalent Keystone Project in subcloud

Reproducibility
---------------
Reproducible, on initial subcloud manage.

System Configuration
--------------------
Distributed Cloud system

Branch/Pull Time/Commit
-----------------------
stx5.0/master
2020-11-27_00-00-09

Last Pass
---------
Did this test scenario pass previously?
Yes. It was observed to pass prior to the introduction of the 'service' project.

Timestamp/Logs
--------------
ERROR in log: Unable to assign role to user on project reference (The project is 'service', not 'services').
2020-12-01 08:03:09.113 106271 ERROR dcorch.engine.sync_services.identity [-] subcoud1/identity: Unable to assign role to user on project reference Resource(id=29,master_id='466f984c13d648c78f18a9ec8d4beb99_5719d84bc8924a3c897bc22056278cde_cbb11e52b38d4f7ab8ca89694bbc3406',resource_type='project_role_assignments',uuid=4fd70497-c7da-4f83-ac0b-fc31a1555fe3):466f984c13d648c78f18a9ec8d4beb99, cannot find equivalent Keystone Project in subcloud

 id | uuid | resource_type | master_id
             | created_at | updated_at | deleted_at | deleted | capabilities
----+--------------------------------------+--------------------------+---------------------------------------------------------------------------------------
-------------+----------------------------+------------+------------+---------+--------------
 29 | 4fd70497-c7da-4f83-ac0b-fc31a1555fe3 | project_role_assignments | 466f984c13d648c78f18a9ec8d4beb99_5719d84bc8924a3c897bc22056278cde_cbb11e52b38d4f7ab8ca
89694bbc3406 | 2020-12-01 08:03:08.895386 | | | 0 |
(1 row)

dcorch=# \q
[sysadmin@controller-0 ~(keystone_admin)]$ openstack project list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 3133dc3d25ac4fb0ae5a1acd4d885dea | admin |
| d582d115a0f64f23b5132b1a191e8864 | services |
| 466f984c13d648c78f18a9ec8d4beb99 | service |
+----------------------------------+----------+

The ‘service’ project was introduced for “Helm v3 and containerized Armada” via https://review.opendev.org/c/starlingx/ansible-playbooks/+/719970
"armada" user and "service" project during bootstrap, in /usr/share/ansible/stx-ansible/playbooks/roles/bootstrap/bringup-essential-services/tasks/bringup_helm.yml

192 - name: Create keystone credentials for armada domain (local host client only)
193 shell: "source /etc/platform/openrc; \
194 openstack domain create {{ armada_domain }}; \
195 openstack project create --domain {{ armada_domain }} 'service'; \
196 openstack user create --domain {{ armada_domain }} \
197 --project service --project-domain {{ armada_domain }} \
198 --password {{ armada_password }} {{ armada_user }}; \
199 openstack role add --project-domain {{ armada_domain }} \
200 --user-domain {{ armada_domain }} --user {{ armada_user }} \
201 --project service admin"

line 195 creates a new project "service", while there is an existing project "services".

Any reason we need a new project instead of using the existing "services" project?

Test Activity
-------------
Developer Testing

Workaround
----------
Wait for 280.002 identity sync alarm to clear.

[Angie Wang (angiewang)]

Armada user, project, and domain are not needed as keystone authentication is not being used in our Armada requests in the application workflow. The decision is made to remove these.

This issue is addressed in commit https://review.opendev.org/c/starlingx/ansible-playbooks/+/741024.
The commit is still on review.

[Angie Wang (angiewang)]

Fixed by: https://review.opendev.org/c/starlingx/ansible-playbooks/+/741024
Merged on 2020-12-15.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794281

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794281

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/163ec9989cc7360dba4c572b2c43effd10306048
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 4e96b762f549aadb0291cc9bcf3352ae923e94eb
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 15:48:19 2021 +0000

    Revert "Restore host filesystems with collected sizes"

    This reverts commit 255488739efa4ac072424b19f2dbb7a3adb0254e.

    Reason for revert: Did a rework to fix https://bugs.launchpad.net/starlingx/+bug/1926591. The original problem was in puppet, and this fix in ansible was not good enough, it generated some other problems.

    Change-Id: Iea79701a874effecb7fe995ac468d50081d1a84f
    Depends-On: I55ae6954d24ba32e40c2e5e276ec17015d9bba44

commit c064aacc377c8bd5336ceab825d4bcbf5af0b5e8
Author: Angie Wang <email address hidden>
Date: Fri May 21 21:28:02 2021 -0400

    Ensure apiserver keys are present before extract from tarball

    This is to fix the upgrade playbook issue that happens during
    AIO-SX upgrade from stx4.0 to stx5.0 which introduced by
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/792093.
    The apiserver keys are not available in stx4.0 side so we need
    to ensure the keys under /etc/kubernetes/pki are present in the
    backed-up tarball before extracting, otherwise playbook fails
    because the keys are not found in the archive.

    Change-Id: I8602f07d1b1041a7fd3fff21e6f9a422b9784ab5
    Closes-Bug: 928925
    Signed-off-by: Angie Wang <email address hidden>

commit 0261f22ff7c23d2a8608fe3b51725c9f29931281
Author: Don Penney <email address hidden>
Date: Thu May 20 23:09:07 2021 -0400

    Update SX to DX migration to wait for coredns config

    This commit updates the SX to DX migration playbook to wait after
    modifying the system mode to duplex until the runtime manifest that
    updates coredns config has completed. The playbook will wait for up to
    20 minutes to allow for the possibilty that sysinv has multiple
    runtime manifests queued up, each of which could take several minutes.

    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/792494
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792496
    Change-Id: I3bf94d3493ae20eeb16b3fdcb27576ee18c0dc4d
    Closes-Bug: 1929148
    Signed-off-by: Don Penney <email address hidden>

commit 7c4f17bd0d92fc1122823211e1c9787829d206a9
Author: Daniel Safta <email address hidden>
Date: Wed May 19 09:08:16 2021 +0000

    Fixed missing apiserver-etcd-client certs

    When controller-1 is the active controller
    the backup archive does not contain
    /etc/etcd/apiserver-etcd-client.{crt, key}

    This change adds a new task which brings
    the certs from /etc/kubernetes/pki

    Closes-bug: 1928925
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: I3c68377603e1af9a71d104e5b1108e9582497a09

commit e221ef8fbe51aa6ca229b584fb5632fe512ad5cb
Author: David Sullivan <email address hidden>
Date: Wed May 19 16:01:27 2021 -0500

    Support boo...