StarlingX

Upgrades: images specified in additional_local_registry_images at install time are not upgraded

Bug #1908100 reported by Yuxing on 2020-12-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Yuxing

Bug Description

Brief Description
-----------------
Images specified in additional_local_registry_images at install time are not upgraded after an upgrade. Currently, there is no option for users to add these additional images as a parameter during a platform
upgrade.

Severity
--------
Minor

Steps to Reproduce
------------------
1. A fresh install of starlingX with additional_local_registry_images specified in localhost.yml in the first controller before bootstrap
2. Upgrade the system using an N+1 version iso

Expected Behavior
------------------
Images in additional_local_registry_images can be upgraded if there is a new version compatible.

Actual Behavior
----------------
Images in additional_local_registry_images cannot get upgrade.

Reproducibility
---------------
100%

System Configuration
--------------------
All systems

Branch/Pull Time/Commit
-----------------------
Master

Last Pass
---------
NA

Timestamp/Logs
--------------
NA

Test Activity
-------------
Developer Testing

Workaround
----------
Manually pull these images from public registries and push them to local registries.

Tags:

Yuxing (yuxing) on 2020-12-14

Changed in starlingx:
assignee:	nobody → Yuxing (yuxing)

Revision history for this message

Yuxing (yuxing) wrote on 2020-12-14:

https://review.opendev.org/c/starlingx/ansible-playbooks/+/767023

Changed in starlingx:
status:	New → In Progress

Ghada Khalil (gkhalil) on 2020-12-15

tags:

added: stx.update

Ghada Khalil (gkhalil) on 2021-01-07

Changed in starlingx:
importance:	Undecided → Medium
tags:	added: stx.5.0

Revision history for this message

Yuxing (yuxing) wrote on 2021-01-07:

https://review.opendev.org/c/starlingx/distcloud/+/769411

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2021-01-07:

stx.5.0 / medium - issue with upgrades

Revision history for this message

Yuxing (yuxing) wrote on 2021-01-14:

commits:

https://review.opendev.org/plugins/gitiles/starlingx/ansible-playbooks/+/587c1ff2075c518f32bb93f14f66bcd23e542259

are merged.

Revision history for this message

Yuxing (yuxing) wrote on 2021-01-14:

Another commit: https://review.opendev.org/plugins/gitiles/starlingx/distcloud/+/43138e6aa59bf057ae4c7d2cda623ed042fc2fc1

is also merged

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

Yuxing (yuxing) wrote on 2021-01-18:

Reopen this LP for fixing a bug introduced by https://review.opendev.org/plugins/gitiles/starlingx/ansible-playbooks/+/587c1ff2075c518f32bb93f14f66bcd23e542259

Changed in starlingx:
status:	Fix Released → In Progress

Revision history for this message

Yuxing (yuxing) wrote on 2021-01-18:

https://review.opendev.org/c/starlingx/ansible-playbooks/+/771271

Revision history for this message

Yuxing (yuxing) wrote on 2021-01-18:

A bug fix is merged: https://review.opendev.org/plugins/gitiles/starlingx/ansible-playbooks/+/c62c3aa8890ef6c0d80ba8397ef64f5b0fb8c291

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-19: Fix proposed to ansible-playbooks (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-19: Fix proposed to distcloud (f/centos8)

#10

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/792298

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-27: Change abandoned on distcloud (f/centos8)

#11

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/792298
Reason: Updated merge soon

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-27: Fix proposed to distcloud (f/centos8)

#12

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/793405

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-02: Fix proposed to ansible-playbooks (f/centos8)

#13

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794286

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-02:

#14

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794288

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-02: Change abandoned on ansible-playbooks (f/centos8)

#15

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794288

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-02:

#16

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794286

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-02: Fix proposed to ansible-playbooks (f/centos8)

#17

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-02: Change abandoned on ansible-playbooks (f/centos8)

#18

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-03: Fix merged to ansible-playbooks (f/centos8)

#19

Download full text (52.5 KiB)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/163ec9989cc7360dba4c572b2c43effd10306048
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 4e96b762f549aadb0291cc9bcf3352ae923e94eb
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 15:48:19 2021 +0000

Revert "Restore host filesystems with collected sizes"

This reverts commit 255488739efa4ac072424b19f2dbb7a3adb0254e.

Reason for revert: Did a rework to fix https://bugs.launchpad.net/starlingx/+bug/1926591. The original problem was in puppet, and this fix in ansible was not good enough, it generated some other problems.

Change-Id: Iea79701a874effecb7fe995ac468d50081d1a84f
Depends-On: I55ae6954d24ba32e40c2e5e276ec17015d9bba44

commit c064aacc377c8bd5336ceab825d4bcbf5af0b5e8
Author: Angie Wang <email address hidden>
Date: Fri May 21 21:28:02 2021 -0400

Ensure apiserver keys are present before extract from tarball

    This is to fix the upgrade playbook issue that happens during
    AIO-SX upgrade from stx4.0 to stx5.0 which introduced by
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/792093.
    The apiserver keys are not available in stx4.0 side so we need
    to ensure the keys under /etc/kubernetes/pki are present in the
    backed-up tarball before extracting, otherwise playbook fails
    because the keys are not found in the archive.

    Change-Id: I8602f07d1b1041a7fd3fff21e6f9a422b9784ab5
    Closes-Bug: 928925
    Signed-off-by: Angie Wang <email address hidden>

commit 0261f22ff7c23d2a8608fe3b51725c9f29931281
Author: Don Penney <email address hidden>
Date: Thu May 20 23:09:07 2021 -0400

Update SX to DX migration to wait for coredns config

    This commit updates the SX to DX migration playbook to wait after
    modifying the system mode to duplex until the runtime manifest that
    updates coredns config has completed. The playbook will wait for up to
    20 minutes to allow for the possibilty that sysinv has multiple
    runtime manifests queued up, each of which could take several minutes.

    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/792494
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792496
    Change-Id: I3bf94d3493ae20eeb16b3fdcb27576ee18c0dc4d
    Closes-Bug: 1929148
    Signed-off-by: Don Penney <email address hidden>

commit 7c4f17bd0d92fc1122823211e1c9787829d206a9
Author: Daniel Safta <email address hidden>
Date: Wed May 19 09:08:16 2021 +0000

Fixed missing apiserver-etcd-client certs

    When controller-1 is the active controller
    the backup archive does not contain
    /etc/etcd/apiserver-etcd-client.{crt, key}

This change adds a new task which brings
the certs from /etc/kubernetes/pki

    Closes-bug: 1928925
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: I3c68377603e1af9a71d104e5b1108e9582497a09

commit e221ef8fbe51aa6ca229b584fb5632fe512ad5cb
Author: David Sullivan <email address hidden>
Date: Wed May 19 16:01:27 2021 -0500

Support boo...

Reviewed:  https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/163ec9989cc7360dba4c572b2c43effd10306048
Submitter: "Zuul (22348)"
Branch:    f/centos8

commit 4e96b762f549aadb0291cc9bcf3352ae923e94eb
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Sat May 22 15:48:19 2021 +0000

Revert "Restore host filesystems with collected sizes"
    
    This reverts commit 255488739efa4ac072424b19f2dbb7a3adb0254e.
    
    Reason for revert: Did a rework to fix https://bugs.launchpad.net/starlingx/+bug/1926591. The original problem was in puppet, and this fix in ansible was not good enough, it generated some other problems.
    
    Change-Id: Iea79701a874effecb7fe995ac468d50081d1a84f
    Depends-On: I55ae6954d24ba32e40c2e5e276ec17015d9bba44

commit c064aacc377c8bd5336ceab825d4bcbf5af0b5e8
Author: Angie Wang <angie.wang@windriver.com>
Date:   Fri May 21 21:28:02 2021 -0400

Ensure apiserver keys are present before extract from tarball
    
    This is to fix the upgrade playbook issue that happens during
    AIO-SX upgrade from stx4.0 to stx5.0 which introduced by
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/792093.
    The apiserver keys are not available in stx4.0 side so we need
    to ensure the keys under /etc/kubernetes/pki are present in the
    backed-up tarball before extracting, otherwise playbook fails
    because the keys are not found in the archive.
    
    Change-Id: I8602f07d1b1041a7fd3fff21e6f9a422b9784ab5
    Closes-Bug: 928925
    Signed-off-by: Angie Wang <angie.wang@windriver.com>

commit 0261f22ff7c23d2a8608fe3b51725c9f29931281
Author: Don Penney <don.penney@windriver.com>
Date:   Thu May 20 23:09:07 2021 -0400

Update SX to DX migration to wait for coredns config
    
    This commit updates the SX to DX migration playbook to wait after
    modifying the system mode to duplex until the runtime manifest that
    updates coredns config has completed. The playbook will wait for up to
    20 minutes to allow for the possibilty that sysinv has multiple
    runtime manifests queued up, each of which could take several minutes.
    
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/792494
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792496
    Change-Id: I3bf94d3493ae20eeb16b3fdcb27576ee18c0dc4d
    Closes-Bug: 1929148
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit 7c4f17bd0d92fc1122823211e1c9787829d206a9
Author: Daniel Safta <daniel.safta@windriver.com>
Date:   Wed May 19 09:08:16 2021 +0000

Fixed missing apiserver-etcd-client certs
    
    When controller-1 is the active controller
    the backup archive does not contain
    /etc/etcd/apiserver-etcd-client.{crt, key}
    
    This change adds a new task which brings
    the certs from /etc/kubernetes/pki
    
    Closes-bug: 1928925
    Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
    Change-Id: I3c68377603e1af9a71d104e5b1108e9582497a09

commit e221ef8fbe51aa6ca229b584fb5632fe512ad5cb
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Wed May 19 16:01:27 2021 -0500

Support bootstrap replay with networking changes
    
    Currently bootstrap playbook replay will fail if the management or
    cluster host networks are changed. To resolve this a couple of changes
    are needed:
    
    * Restart the sysinv agent and wait until it is ready. The sysinv agent
      uses the current management ip for the rabbitMQ connection and
      applying runtime manifests. The process needs to be restarted to
      resync that data.
    
    * Copy the etcd certs to the /opt/platform on replay. The etcd-server
      certs are regenerated on replay. When the cluster host network changed
      the SAN in the certs under /opt/platform were out of date resulting in
      kube-apiserver failures on controller-0 unlock.
    
    Closes-Bug: 1925668
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>
    Change-Id: I228321a2540a0024cd217ed844feb54be9ae3b29

commit 41ada83e4f4486d0795eed3e7a8bbe4227ee88d8
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Wed May 19 12:53:26 2021 -0500

Bug fix: update barbican external id if the project id changes
    
    The previous commit
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/790824
    unexpectedly moves the update_barbican_project_external_id out of the
    scope of the project id changes. Further, the old_id['id'] is
    duplicated with the current_project_id, it will be undeclared in the
    case of the project id not changed (by deleting the subcloud in
    powered off status and re-add it with --migrate option). This commit
    deletes the duplicated variable and moves the
    update_barbican_project_external_id to the correct scope.
    
    Tests:
    1 Delete a subcloud from a central cloud and add it back with the
    migrate option.
    2 Migrate a subcloud successfully to a different central cloud.
    3 Migrate a subcloud successfully with an extra nonlocal user to a
    different central cloud.
    
    Partial-Bug: 1928139
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: I8acf7bcea5d8e77b92877427ccd40b95e8b3515e

commit ac0c5d51a8708c8c056a736ff11ce1a0b1550c4f
Author: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date:   Thu May 13 19:53:41 2021 +0300

Create the pod_max_pids service parameter
    
    This adds a default entry to service parameters.
    
    Create the default entry taking into consideration the most hungry of
    the optional StarlingX apps. The user is free to modify the value as
    desired, using 'system service-parameter-modify'.
    
    Same can be created by the user using 'system service-parameter-add',
    but this helps the user by being transparent in service-parameter-list.
    
    If this service parameter was missing an entry, then no hieradata
    variable would have been generated, so puppet would have used
    a predefined value.
    
    Partial-Bug: 1928353
    Depends-On: I74fcf2bd405c2a3811a4f27a55b28c0d001430e1
    Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
    Change-Id: I707ddc4ca67595fbf809c6ffc15ecd4fb21f4661

commit 64bf73c85c5de0737c8a1cf967b6b251288ee424
Author: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date:   Tue May 11 16:22:12 2021 +0300

Enable kubelet support for pod pid limit
    
    This protects the system before the unlock. This has the most meaning
    during the restore procedure, when the system is unprotected until
    unlock (until puppet generates the config file containing protection).
    
    Partial-Bug: 1928353
    Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
    Change-Id: I09c4d4f494bc113ae8b439256655476e03b54b0e

commit cfc719b82a6f1651a2b3950b316244f907d58491
Author: Angie Wang <angie.wang@windriver.com>
Date:   Mon May 17 17:11:12 2021 -0400

Configure kubeadm to not apply the default taint
    
    The taint "node-role.kubernetes.io/master:NoSchedule" needs
    to be removed from master node so that pods can be scheduled
    on it. This is handled by a bootstrap task. However, issue
    was seen that the default taint was not removed during bootstrap
    that causes armada pod fails to be scheduled on controller-0.
    This happens on one of the subcloud when bootstrapping a batch
    of 50 subclouds.
    
    Add configuration in kubeadm to not apply the default taint
    at the beginning so it doesn't need to be removed afterwards.
    
    Tested AIO-SX, DX upgrade and a batch of 50 subclouds deployment
    
    Change-Id: I543280ddd55ec94ccf0586dc07877349baa06bdd
    Closes-Bug: 1928722
    Signed-off-by: Angie Wang <angie.wang@windriver.com>

commit 45c74db36670e9cba3475e598a4655490b744cee
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Thu May 13 10:46:02 2021 -0500

Cleanup of subcloud rehoming playbook
    
    As the ansible log is accessible for all the users, we don't want to
    expose the keystone IDs and passwords in a subcloud's ansible log.
    This commit hides the keystone IDs and passwords in the ansible log.
    
    This commit also removes un-used facts and the rehome_in_progress
    flag.
    
    Tested with successfully re-homed a SX and a DX subcloud in virtual
    box env.
    
    Story: 2008774
    Task: 42462
    
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: I6c2e83ba5b4923e9d7c82ccf94165608739e59e1

commit 0ca19e0870e06b601553ea2a9d9e1cfc0367d75f
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Tue May 11 16:02:49 2021 -0500

Clean non-local keystone user during migrating keystone IDs
    
    During the progress of re-homing a sub cloud to a new central cloud,
    insert a new user ID will be failed if there's a duplicated non-local
    user in keystone database. This commit deletes the non-local user with
    the same user id before inserting a new keystone local user into the
    database.
    
    Test:
    1. Add a non-local user in keystone database, and this user should
    have a same user ID as the new central cloud.
    2. Successfully re-home this subcloud to the new central cloud.
    3. Successfully re-home a subcloud without the duplicated user in
    keystone database.
    
    Partial-Bug: 1928139
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: Iecaac3400cb9362acc63686ae9470cf01f7eb2e1

commit 275e046280a71b0c5c8c637fdc7065d4e2e19aee
Author: Isac Souza <IsacSacchi.Souza@windriver.com>
Date:   Thu May 13 13:07:56 2021 -0300

Fix deletion of temp dir when backup fails
    
    When the /opt/backups disk is full and the backup playbook
    is executed, the Ansible task fails to create a temp directory.
    After the failure the cleanup code is executed but if the temp
    dir was not created, it fails because tempdir.path will be undefined.
    This second failure aborts the cleanup code and the
    "System Backup in progress." will stay on indefinitely, requiring
    manual intervetion.
    
    Tested by executing the backup procedure when /opt/backups
    is full.
    
    Closes-Bug: 1928365
    Signed-off-by: Isac Souza <IsacSacchi.Souza@windriver.com>
    Change-Id: I76c852b65bde4b40a22bcb9be3a81776ade86d15

commit 02b36df15ba42c4346b34b4ddce29d95b4c7fc69
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu May 13 12:17:06 2021 -0400

Enable ssh connection retries in Ansible playbooks
    
    In this commit:
     - Allow ssh retries in the playbooks in case of unreachable
       failures that are observed in batch subcloud deployment or
       upgrade where the system controller has low network bandwidth.
     - Increase ssh timeout to accomodate slow sudo response when
       ldap service is not yet available for the subcloud.
     - Allow docker login and images pull retries in case docker
       or docker proxy throws an exception due to slow response
       (e.g. oam network is overwhelmed).
    
    Closes-Bug: 1928357
    Change-Id: Ibc6155671b20a01340b66270c3c402174d34ab9e
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 20e44c6dd71758f89bbfe88c6204e777c17deb5d
Author: Zhipeng Liu <zhipengs.liu@intel.com>
Date:   Tue Apr 27 01:40:16 2021 +0800

Fix bootstrap replay failure when changing mgmt subnet
    
    After mgmt subnet is changed, we use previous controller_0
    address for etcd puppet apply to avoid resycing an nonexistent
    hieradata file.
    
    Change-Id: Ie31c48153af30df240237013dd51bfffea5213cd
    Closes-Bug: 1925668
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

commit 7228f60e146286d0948ac67bee159a7b8c54f704
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Tue May 11 18:56:03 2021 +0300

Fix restore user images playbook
    
    When running restore_user_images playbook,
    after the images will be uploaded to the local registry,
    the docker images stored in the local docker filesystem
    will be deleted by a python script called by the playbook.
    Sometimes this script will try to delete images
    that are not present into the local docker filesystem and
    will throw some errors.
    We fix this by adding a check before trying to delete an image.
    
    Also, updated the
    /playbooks/roles/common/push-docker-images/files/download_images.py
    file to do the same checks since it does similar steps
    and can also fail when deleting
    the images from the local docker cache.
    
    Closes-Bug: 1928092
    Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>
    Change-Id: I1be45b2669ad1ab209aa49f050106dd1a7759cee

commit d5460198dc0310a80580537fd8df76ae00e17f02
Author: Robert Church <robert.church@windriver.com>
Date:   Wed May 12 22:45:38 2021 -0400

Adjust armada's tiller container liveness probe
    
    With the liveness probe update in the armada helm chart to test the
    connectivity to the postgres backend, adjust the periodSeconds and
    failureThreshold to align with the minimum swact time to be expected for
    postgres switching from one controller to another.
    
    Reviewing logs from various H/W labs it appears that average postgres
    swact time ranges from 9s-20s, with the mean ~15s.
    
    Times can be observed with:
    2021-05-09T13:32:24.475 controller-1 OCF_pgsql(postgres)[396293]: info
                                         INFO: server shutting down
    2021-05-09T13:32:33.423 controller-0 OCF_pgsql(postgres)[147541]: info
                                         INFO: server starting
    
    Set the periodSeconds to 4 and the failureThreshold to 2 so that if the
    postgres server is not accessible, the tiller container will be
    restarted within the 9s minimum swact time. This will ensure that the
    next time tiller is required by Armada or used by the helmv2-cli that
    the connection to postgres backend has been re-established.
    
    Change-Id: I7454a737771d9a608d2fe69c5136d37da022007e
    Depends-On: https://review.opendev.org/c/starlingx/integ/+/791092
    Related-Bug: #1917308
    Signed-off-by: Robert Church <robert.church@windriver.com>

commit 9d4203b7e0d4b321fcccf14e69367573ef5d5093
Author: Tao Liu <tao.liu@windriver.com>
Date:   Wed May 12 20:30:37 2021 -0400

Use async_timeout for kubectl wait timeout
    
    Data migration failed for 5 subclouds during 30 subcloud upgrade.
    This failure was caused by timeout waiting for Kubernetes component,
    Networking or Armada pods to be ready.
    
    The async_timeout is set to 120s in restore mode, but the pod wait time
    is still set to 30s. This update changes the pod wait time to be the
    same as async_timeout.
    
    Tested on DC-2 with 50 subcloud parallel upgrade
    
    Closes-Bug: 1928252
    
    Signed-off-by: Tao Liu <tao.liu@windriver.com>
    Change-Id: If2b21d0cd2e0de9e84869323a43d4a249d031132

commit 36451c99ce76e76084ad5c68e4954bf347e8c0b7
Author: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date:   Tue May 11 16:24:26 2021 +0300

Add helm sql database ip to armada overrides
    
    This will be used by tiller container to check that the container
    networking is properly set up.
    
    Partial-Bug: 1928141
    Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
    Change-Id: I177bb628497611eb64472291a04d635856c26590

commit 86fbafec14744d6630aff6e6ac2bb165c5be2be8
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue May 11 14:58:29 2021 -0400

Fix: Upgrade does not replay correctly after activate failure
    
    When the etcd runtime manifest fails during upgrade activate,
    the activation fails as expected. However when the upgrade-activate
    is attempted again, the etcd upgrade playbook is not re-run.
    Depending on when the manifest fails, this could result in secure
    etcd remaining disabled on the system.
    
    This commit makes it possible to replay the etcd upgrade playbook,
    thus solving the issue.
    
    Change-Id: I7f453d9040916381519ac96ed4567ec5fb6e7a8d
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1928130

commit 2dfadea581d1ac9955a7921bd2730e36a152255c
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Thu Mar 11 11:12:05 2021 -0500

Add a rehome subcloud playbook
    
    This commit creates a rehome_subcloud playbook to migrate an existing
    subcloud from the original central cloud to a new central cloud. This
    playbook is expected to be remotely play from the new central cloud
    system controller based on the subcloud's overrides. This playbook
    updates the system controller network info in the subcloud, migrates
    the keystone data and updates the subcloud's ca and certs. After
    running this playbook and unlock the subcloud controllers, the
    subcloud can be discovered online in the new central cloud, and
    can be managed and brought in-sync from the new central cloud.
    
    Test:
    Successfully migrate a AIOSX and a AIODX subcloud to the new central
    cloud with this playbook.
    
    Depends-On: https://review.opendev.org/c/starlingx/config/+/784767
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/785977
    Depends-On: https://review.opendev.org/c/starlingx/config/+/786638
    Depends-On: https://review.opendev.org/c/starlingx/config/+/787213
    
    Story: 2008774
    Task: 42152
    Change-Id: Iaa6699951e855c76602bd43f71d42e64e298c786
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit d23b932208b778ec73cc51bccf460310f16ceaad
Author: Bin Qian <bin.qian@windriver.com>
Date:   Mon May 3 15:32:06 2021 -0400

Ensure n3000-opae image cache is not deleted
    
    Skip removing n3000-opae image cache. This image is expected to be
    available to reset n3000 fpgas before docker local registry is ready.
    
    Closes-Bug: 1927000
    Signed-off-by: Bin Qian <bin.qian@windriver.com>
    Change-Id: I3372d89c48b394c8cac6b9da06d2528bb6afa803

commit 255488739efa4ac072424b19f2dbb7a3adb0254e
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Thu Apr 29 16:37:21 2021 +0300

Restore host filesystems with collected sizes
    
    Since https://review.opendev.org/c/starlingx/ansible-playbooks/+/784860,
    the host filesystems(backup, docker, kubelet, scratch) are
    no longer resized in ansible at restore and they are not using the
    collected sizes from the backup archive. Puppet will try to
    resize them when unlocking but this will generate some errors.
    
    The solution is to create the host filesystems with the
    correct sizes at restore. The sizes are taken from the
    backup archive.
    
    Closes-Bug: 1926591
    Change-Id: Id670408a518e4a1e3fc75a668eea42d26a972d66
    Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>

commit bc0fba6bbbd0182c4886e5a3ccbfc2d0973cfd70
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Apr 28 11:52:36 2021 -0400

Remove restore subcloud admin endpoint certificate from config
    
    This change is to exclude the admin endpoint certificate from restore
    to config directory. The admin endpoint certificate is stored in k8s
    (backup) and restore as part of k8s restore. Sysinv will generate it
    into hieradata from k8s secret and puppet will genereate the pem for
    haproxy.
    
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/786666
    Partial-Bug: 1923510
    Signed-off-by: Bin Qian <bin.qian@windriver.com>
    Change-Id: Iae8fb9c53e0aa6797a25b872adb0c99636c4243a

commit e88c9290a3ca2d947828a1f4e988f9fe61a1a623
Author: Melissa Wang <melissa.wang@windriver.com>
Date:   Mon Apr 26 17:45:40 2021 -0400

SX to DX migration: Check network interface config
    
    This change adds a semantic check to ensure that the cluster-host
    and management networks are not configured on loopback before
    allowing sx-to-dx migration.
    
    Task: 42375
    Story: 2008587
    
    Change-Id: I87326db222ffe9eb8bf23b69d17f676abc7c242d
    Signed-off-by: Melissa Wang <melissa.wang@windriver.com>

commit e25439d49d127779f9ab32650a4a51027242884b
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Apr 14 14:53:56 2021 -0400

Remove creating admin endpoint cert in subcloud bootstrap
    
    This change removes creating admin endpoint cert in subcloud
    bootstrap.
    The admin endpoint cert is generated in manifest at the time when
    the controller node is unlock the first time. The cert data is
    retrieved directly from k8s secret data (where cert-manager is
    responsible to maintain it and keep it up to date).
    
    Partial-Bug: 1923510
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/786666
    Signed-off-by: Bin Qian <bin.qian@windriver.com>
    Change-Id: Ie6a5c8fe159efcdebdb4c81666e981772408b82c

commit 5b286734637ca6f503a62131b309829d2f308fed
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Mon Apr 5 17:01:48 2021 -0500

Remove drbd resize actions from bootstrap playbook
    
    drbd resize actions can cause failures during unlock. Avoid all drbd
    resize actions during the bootstrap playbook. Pass the correct
    filesystem sizes to the bootstrap manifest. Use the collected drbd
    filesystem sizes during the restore/upgrade bootstrap manifest.
    
    Closes-Bug: 1920245
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>
    Change-Id: I235575ad40ba84298d3db5b39ce7861a143c78a8

commit 05d3d8c21f5017b26466bfd1cdd1f1e7accf266f
Author: Carmen Rata <carmen.rata@windriver.com>
Date:   Mon Apr 12 10:35:57 2021 -0400

Fix kube-apiserver pod removal
    
    This commit adds rolebinding configuration to bind
    the "privileged-psp-user" role to the kubelet user.
    It fixes the issue where the kube-apiserver pod does
    not get recreated after enabling PodSecurityPolicy
    plugin. With this fix we make sure to allow the apiserver
    pod creation by granting permission to the kubelet user
    to create that pod.
    
    Closes-Bug: 1881605
    
    Signed-off-by: Carmen Rata <carmen.rata@windriver.com>
    Change-Id: Ibdf6d4cacf2ce83dfa744455dac460027b2a6e47

commit 802afc1c23d26f44bd7954a5ec038c5badddcaa3
Author: Andrei Grosu <andrei.grosu@windriver.com>
Date:   Mon Feb 8 22:28:49 2021 +0000

Inform conductor of ansible backup actions.
    
    Let apps run custom code around backup actions inside the playbook.
    
    Story: 2007960
    Task: 40769
    Depends-On: I0ebab45f4846cbcd25fecac6bf99195d9047eb8a
    
    Signed-off-by: Andrei Grosu <andrei.grosu@windriver.com>
    Change-Id: I61156db05970aa03c96ddc8533fdd4f4a680b334

commit f6cd32b82bc01c9f1b9f53d1754aac6a45e51643
Author: Don Penney <don.penney@windriver.com>
Date:   Fri Mar 26 11:23:51 2021 -0400

Copy default-registry-key to deployment namespace
    
    After creating the deployment namespace, copy the default-registry-key
    from the kube-system namespace into it for use by pods running in that
    namespace.
    
    Story: 2007361
    Task: 42186
    Signed-off-by: Don Penney <don.penney@windriver.com>
    Change-Id: I3b431c74295cc0099f00814ee28709b1b4c56c8c

commit d61c82f555034f104e1bee8a83bb19ad448012b2
Author: Zhipeng Liu <zhipengs.liu@intel.com>
Date:   Wed Mar 31 23:12:43 2021 +0800

Change CN of etcd CA since etcd will reuse kubernetes CA
    
    Basic test pass on simplex.
    
    Closes-Bug: 1921511
    
    Change-Id: If3a7cca4a03b05ac5eb61f7f579449a7393c1644
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

commit 8a7c871a8d7c8843b404e6779eccd2e483293c2a
Author: Carmen Rata <carmen.rata@windriver.com>
Date:   Mon Mar 22 15:32:48 2021 -0400

Add default psp configuration to psp-policies.yaml
    
    Added ClusterRoleBindings configuration in psp-policies.yaml to
    enhance the current capabilities.
    The new configuration gives users the ability that by default,
    they can create pods and deployments with “restricted” capabilities
    in their namespaces.
    When psp policies are applied at bootstrap time, all tenants/users
    have at least restricted capabilities and access rights will be
    added as needed, reducing this way the manual configuration
    required.
    
    Closes-Bug: 1885716
    
    Signed-off-by: Carmen Rata <carmen.rata@windriver.com>
    Change-Id: I6118efd6832836829d31aa84b2b4305d5a1f24c4

commit f7d5491e404ba2854105278f6b9e2883b52a5206
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Aug 25 12:16:22 2020 -0400

Remove CPU resource requests from platform pods
    
    In order to allow applications to make use of all "application" cores,
    we need to remove all CPU resource requests from "platform" pods.  We tried
    to make this work by modifying Kubernetes itself to track platform pod resources
    separately, but it would have required changes to kube-scheduler, which would
    involve building a custom kube-scheduler container image.
    
    Some platform pods are created by "kubeadm" during early init and are handled
    via a kubernetes code change.  For calico, multus, and sriov we modify the
    templates used to create the pods and remove any reference to CPU resources.
    
    Normally this would mean that they could get throttled by application pods, but
    since they're running on the platform cores this shouldn't be a problem.
    
    Story: 2008760
    Task: 42170
    Change-Id: Ibf73bd0d105e4040f02a4114afbb31d131fc9585
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>

commit 41de2e52db9985d84397d7ba56a59bbeaa9cf88f
Author: Zhipeng Liu <zhipengs.liu@intel.com>
Date:   Fri Mar 19 01:00:10 2021 +0800

Fix bootstrap replay fails due to not changing etcd config.
    
    When replay bootstrap, we still need to apply etcd puppet manifest.
    Test PASS for below cases
    1）System bring up with bootstrap replay
       - non cluster host network reconfiguration
    2）System bring up with bootstrap replay
       - cluster host network change
    3）AIO simplex backup and restore
    
    Closes-Bug: 1918943
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>
    Change-Id: Ib892776f4b2949f9255fc4725add1f0e362956a9

commit fbd6a848d53ae6de9703466215fb85df2485ab0b
Author: Scott Little <scott.little@windriver.com>
Date:   Thu Mar 25 11:35:34 2021 -0400

Set SW_VERSION 21.05
    
    Prep for the StarlingX 5.0 release.
    SW_VERSION uses YY.MM format.
    
    Story: 2008055
    Task: 42113
    Depends-On: https://review.opendev.org/c/starlingx/utilities/+/783042
    Signed-off-by: Scott Little <scott.little@windriver.com>
    Change-Id: Ic718ed214dfdb6eb8fcfffcadbbeadfbb8f6d052

commit 2ffd9c4bff6bd736f35cc1fb2fd3b0e25c9ef8f2
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Fri Mar 19 10:36:55 2021 -0500

drbd filesystems not resized during bootstrap
    
    Remove drbd resize_result check on resize2fs operation. Both operations
    should run when requested. These commands will return 0 when the disk is
    resized and will return 0 if the disk is already correctly sized. Any
    non-zero return code should fail the playbook.
    
    Closes-Bug: 1920245
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>
    Change-Id: I946f63b2886b5377494e658a59586005c27ec2d2

commit 8c2580cc85b8e7a2783ece895d7c2c476db81990
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Mar 10 12:13:56 2021 -0500

Introduce SX to DX migration playbook
    
    This commit introduces a migrate-subcloud.yml playbook that the user
    can run, with an overrides file to provide config values, to perform
    the migration steps for a subcloud. Once the migration playbook has
    been applied and the subcloud has recovered from the unlock (performed
    by the playbook), the second controller can be installed and
    configured.
    
    Story: 2008587
    Task: 41743
    Depends-On: https://review.opendev.org/c/starlingx/config/+/776536
    Signed-off-by: Don Penney <don.penney@windriver.com>
    Change-Id: I1d8c1219694147baaabb183ef3debe1715aaf153

commit 64feca44b59284b07d42e7100fb90b22297c0ce0
Author: Carmen Rata <carmen.rata@windriver.com>
Date:   Fri Mar 5 15:18:46 2021 -0500

Use jinja2 template for containerd config.toml
    
    File "/etc/containerd/config.toml" is set up with ansible bootstrap
    using an erb template managed by puppet.
    We replace the erb template with a jinja2 template to improve ease
    of use and reduce the need for complex regular expressions.
    
    Closes-Bug: 1892768
    Signed-off-by: Carmen Rata <carmen.rata@windriver.com>
    Change-Id: I93601321d4f554d27bb9457bcff99428351cbefd

commit df1f8e381d951a2ab6c52a32eb2467817795e083
Author: Suvro Ghosh <suvrojeet.ghosh@windriver.com>
Date:   Thu Mar 11 10:38:57 2021 -0500

Adding force flag to purge task
    
    This flag will allow ansible replay
    
    Story: 2007960
    Task: 42039
    Signed-off-by: Suvro Ghosh <suvrojeet.ghosh@windriver.com>
    Depends-On: If68d66d799addcd996da4b146d092c855b455aa3
    Change-Id: I93821965184d95a00fddd3398a1c214e3d730efa

commit ed7314d4c4c1f9d6f18dc6c144dd4b6cdf67b66e
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Mon Mar 8 14:36:54 2021 +0200

Fix bootstrap playbook when initializing kubernetes
    
    The etcd endpoint in the kubedeadm file is different than
    the endpoint defined in the etcd config file.
    This is because in the kubeadm file, the etcd endpoint is equal to
    the 'cluster_floating_address' variable. And in the etcd config
    file the 'default_cluster_host_start_address' variable is used.
    These 2 variables can be different when the
    'cluster_host_start_address' variable defined in the localhost.yml
    differs from the first address of the 'cluster_host_subnet'.
    
    The solution is to use 'cluster_floating_address' in both cases
    because this variable is defined in the following way:
    
    cluster_floating_address: "{{ address_pairs['cluster_host']['start'] }}"
    will be different than the 'default_cluster_host_start_address': "{{
    (cluster_host_subnet | ipaddr(1)).split('/')[0] }
    
    So it will use 'default_cluster_host_start_address' when the
    'cluster_host_start_address' is not defined.
    
    Closes-Bug: 1918130
    Change-Id: I8fecc1e5e54b5a9a9a72a54c069f79f5f2d434ba
    Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>

commit ff5c9327c1990cb2bf6454a1a156669187dc056b
Author: Babak Sarashki <babak.sarashki@windriver.com>
Date:   Wed Feb 24 05:45:30 2021 +0000

Remove container runtime interface (CRI) placeholder
    
    Change-Id Ib1dd5bd (stx-puppet) and Icc5fd16 (config) add
    support to set CRI entries for kubernetes runTimeClass.
    
    This is not needed during the initial bring-up of services
    at bootstrap time and produces incorrect config.toml causing
    bootstrap failure. Therefore, it is removed from the initial
    config.toml generated during bring up of essential services.
    
    Story: 2008434
    Task: 41928
    
    Depends-On: https://review.opendev.org/c/starlingx/config/+/776220
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/776223
    
    Signed-off-by: Babak Sarashki <babak.sarashki@windriver.com>
    Change-Id: Id124653329da0ba9f990e5bba6b53faa3c88fa86

commit ae898b781539f71ea009cefec7602ed62335741b
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Feb 11 13:22:39 2021 -0500

Create device_images bind directory
    
    The device images are stored in the drbd filesystem
    (/opt/platform/device_images) in the active controller.
    In order to allow the other worker hosts to retrieve the device images
    from the active controller over lighttpd, the directory
    /www/pages/device_images is created as a bind mount of the drbd
    directory.
    
    Tests performed on the following systems:
    AIO-DX, AIO-DX plus compute, Standard 2+1
    DC with AIO-DX plus subcloud
    DC with Standard subcloud
    
    Story: 2007875
    Task: 41878
    
    Change-Id: I00c75767543d3840c466df887e9f16ba75a5386d
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit 9814ec0490506f222326396368f34e278ae52a0d
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Feb 22 23:07:34 2021 -0500

Update Ansible bootstrap address validation
    
    The method to validate bootstrap address in previous
    commit (9c62c83536b737e731b140c95be37d74769989ff) is not
    reliable for IPv6. This commit fixes it.
    
    Task: 41800
    Story: 2008573
    Change-Id: Ibadf36e7f6c1ec31ca47514802991c92959fd138
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 9cee892563c68154714bcb7a7e173dcc13b6b237
Author: Angie Wang <angie.wang@windriver.com>
Date:   Wed Feb 24 14:27:14 2021 -0500

Update the minimum root disk size to be aligned with sysinv
    
    Change-Id: I03c8b31ab76ce8a2b8534677910763150ec1d9c0
    Closes-bug: 1916797
    Depends-On: https://review.opendev.org/c/starlingx/config/+/777465
    Signed-off-by: Angie Wang <angie.wang@windriver.com>

commit f2d20c15bbda992bec707b9dd3529f5f1fd53b83
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Tue Feb 2 18:09:33 2021 +0200

Fix restoring dc-vault on a central controller
    
    At this moment, when we do a restore procedure on a
    DC system controller, the /opt/dc-vault directory will be
    created under "/" filesystem. It should be created on
    a separated filesystem, but that filesystem is available
    only after an unlock of the controller.
    
    The proposed solution is to create an additional restore
    playbook for the dc-vault that will be manually run after
    unlocking controller-0. The backup playbook will create
    an additional archive with the contents of dc-vault, and
    the dc-vault directory will be removed from the platform
    backup.
    
    The new playbook will be used like this:
    
    ansible-playbook
    /usr/share/ansible/stx-ansible/playbooks/restore_dc_vault.yml -e
    "ansible_become_pass=Li69nux*" -e "admin_password=Li69nux*" -e
    "initial_backup_dir=/home/sysadmin" -e
    "backup_filename=localhost_dc_vault_backup_2021_02_02_11_46_09.tgz"
    
    Closes-Bug: 1914258
    Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>
    Change-Id: I8fdd5b678e2296cd0ce98ea4dd91e2988beb200f

commit 3babc1eed3ba861c652f082655fa284992be0859
Author: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date:   Fri Feb 19 16:45:22 2021 +0200

B&R: Fix registry backup generated when it should not
    
    Incorrect ansible variable evaluation results in the a backup file
    generated when it is required not do so.
    
    Fix the evaluation.
    
    Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
    Depends-On: I4644784ea4164134f163d218e69dc4ceb148985a
    Closes-Bug: 1916246
    Change-Id: I2a31dcda55137a668b2e82b9a938535bdf623656

commit 151b885dc733d5094fce474ae042ee2f2ceae49e
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Fri Feb 19 16:21:23 2021 +0200

B&R: Fix backup hangout on IPv6 systems
    
    IPv6 addreses must be enclosed in square brackets when they are
    followed by a port number.
    
    At the "[backup/backup-system : Create etcd snapshot]"
    step, the etcd IPv6 endpoint is not wrapped in square brackets,
    so the command hangs indefinitely.
    
    We fix this by using the 'ipwrap' ansible filter which will
    wrap the address in [] brackets if it's an IPv6 one.
    
    Closes-bug: 1916053
    Change-Id: If40ed59f4e44c9f877aaefe87f6211a3e83ddfee
    Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>

commit 71db4f1aa82a36c360ada97a6d8d989a736e1133
Author: Steven Webster <steven.webster@windriver.com>
Date:   Wed Feb 17 13:54:14 2021 -0500

Uprev SR-IOV CNI image
    
    This commit uprevs the SR-IOV CNI image to pick up a few bug
    fixes.  Specifically, this commit will allow rate-limiting
    configuration on a VF to be retained after the VF has been
    used by a pod (and pod subsequently deleted).
    
    Testing:
    
    NICs:
    Ethernet Controller X710 for 10GbE SFP+
    Mellanox MT27700 Family [ConnectX-4]
    
    Functional:
    Connectivity testing (kernel + DPDK)
    Devices allocated appropriately to pod
    Rate-limiting information retained after pod deletion
    
    Closes-Bug: #1915951
    
    Signed-off-by: Steven Webster <steven.webster@windriver.com>
    Change-Id: Ie893fee61fa15e28e1994b5e766ed3bca2ff4050

commit 9c62c83536b737e731b140c95be37d74769989ff
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Tue Feb 9 23:08:32 2021 -0500

Ansible update to support remote subcloud restore
    
    In this commit:
      - Common code to validate target is updated to include additional
        host check options (system readiness, load version, bootstrap ip,
        and patches).
      - A new playbook is added to perform host check for various
        use cases (remote install, pre subcloud restore, pre subcloud
        upgrade, etc...).
      - Install playbook is updated to make use of the new playbook.
      - A new role which performs generic user input validation for all
        restore playbooks is added.
      - A new B&R parameter is added to indicate where the backup data
        can be found, on the host itself (on-box) or on another
        machine (off-box).
      - Platform, user images and openstack restore playbooks are
        updated to a) make use of the on_box_data parameter,
        b) use the same target_backup_dir for both local and remote
        playbook execution for consistent behavior.
      - Host override file is extracted only on the target.
      - Patches restore is skipped if requested by the caller. Default
        behavior is to restore patches.
      - Various subtle bugs are fixed.
      - Ansible version is specified in Zuul test requirements.
    
    Tests:
      - Deployment of a Redfish capable subcloud
      - Remote restore a simplex with various options: a) without patches
        b) skip patches restore, c) with patches restore, d) on-box backup
        tarball, e) off-box backup tarball
      - Local restore of a simplex with 2 options: a) without patches and
        b) skip patches restore
      - Simplex fresh install
      - Restore user images
      - Restore OpenStack
      - Error cases
    
    Task: 41725
    Story: 2008573
    Change-Id: Ica2b9010a73854a01216e2e16b581484d182264e
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 64af997f72d2e78615bd38d573aa13d0a59932da
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Thu Feb 4 17:55:24 2021 -0500

Fetch additional image config file for remote play
    
    When invoking the playbook remotely, the additional system images
    config file locates in the remote host may not also exist in the
    control host. This commit fetches the additional image config file to
    the control host to prevent include_vars failure.
    
    Tests:
    1 Remote backup and restore an AIOSX node with the additional system
    images config file from a control host without the config file.
    2 Build an image ISO and installed/ bootstrapped an AIOSX node.
    
    Closes-Bug: 1914611
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: I79362b00a25ca031a1fbbaa92476955e35477b73

commit 6226fa7a28ee271f92419bbeb62f2ee512e4b192
Author: Mingyuan Qi <mingyuan.qi@intel.com>
Date:   Tue Dec 8 05:52:41 2020 +0000

Add provision edgeworker playbook
    
    This playbook provisions edgeworker nodes to setup OS
    configurations as well as join the node to the Kubernetes
    platform.
    
    This playbook should be triggered manually after an edgeworker node
    added to sysinv. Make sure the edgeworker node has got the correct
    mgmt ip address from controller's dhcp server.
    
    1. Create an inventory file for edgeworker nodes on the active
    controller:
    
    tee ./edgeworker_inventory.yml << EOF
    all:
      hosts:
        localhost:
          ansible_connection: local
      children:
        edgeworker:
          hosts:
            <edgeworker hostname>:
              ansible_ssh_user: <admin username>
              ansible_ssh_pass: <ssh password>
              ansible_become_pass: <admin password>
              ansible_python_interpreter: <python path(e.g.
    /usr/bin/python3)>
            #<more edgeworker nodes>:
      vars:
        ansible_ssh_user: sysadmin
        ansible_ssh_pass: St8rlingX*
        ansible_become_pass: St8rlingX*
    EOF
    
    2. Trigger the playbook provision_edgeworker.yml:
    
    ansible-playbook \
    -i ./edgeworker_inventory.yml \
    /usr/share/ansible/stx-ansible/playbooks/provision_edgeworker.yml
    
    After the provisioning, the edgeworker node will be shown Ready in
    Kubernetes.
    
    Test:
    Edgeworker node OS: Ubuntu 18.04/20.04
    AIO-SX + edgeworker nodes: PASS
    AIO-DX + edgeworker nodes: PASS
    Standard + edgeworker nodes: PASS
    
    Story: 2008129
    Task: 40879
    
    Signed-off-by: Mingyuan Qi <mingyuan.qi@intel.com>
    Change-Id: If189f915461da3ce79dea3a688a3cf5e59d8f12c
    Depends-On: https://review.opendev.org/c/starlingx/config/+/763918

commit 2a5e37cecfde7572331b0c7f13e8ea55400c5260
Author: Douglas Henrique Koerich <douglashenrique.koerich@windriver.com>
Date:   Mon Feb 1 09:26:48 2021 -0500

Remove RPF config of tunl0 interfaces from kernel
    
    Calico project removed "strict" mode for RPF from kernel's IPv4
    configuration at version 3.12, moving back to former approach of
    setting iptables rules for security purposes (see
    https://github.com/projectcalico/felix/pull/2189). Thus, forcing
    disable of RPF at tunl0 interface as introduced by the fix of
    bug 1838801 isn't necessary anymore.
    This commit removes that forced writing of kernel configuration,
    so that calico-node pod is healthy after started, and asymmetrical
    routing still works as required.
    
    Implements: removal of postStart hook for Calico pods
    Closes-Bug: 1912807
    Signed-off-by: Douglas Henrique Koerich <douglashenrique.koerich@windriver.com>
    Change-Id: Iea8f4aa988269f99f5632fec2cf344297709eaef

commit adf542c16b71e681e47599b1e60a17311efeb6a6
Author: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date:   Thu Jan 21 09:05:22 2021 +0200

B&R: Fix restore of ceph using mds
    
    Calling /etc/init.d/ceph without specifying a component applies an
    action to all components.
    After cephfs_platform_integ topic merged, ceph mds component is also
    used.
    Mds requires a monitor to be up before starting.
    Due to the fact that the ceph script was called without specifying the
    components mds the start action was applied to mds also and it was
    applied before starting the monitor, resulting in failure.
    
    The fix is to specify the monitor and osd components.
    Mds will be start when ceph is restarted a few tasks later.
    
    Plus minor ansible changes:
    - Use failed_when instead of ignore_errors
    - Renamed task
    
    Closes-Bug: 1912488
    Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
    Change-Id: I5388ddd32923bbd9acb205fa2d2adbc083346631

commit c62c3aa8890ef6c0d80ba8397ef64f5b0fb8c291
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Jan 18 15:49:21 2021 -0500

Bug fix: static image type error
    
    A bug was introduced by commit
    587c1ff2075c518f32bb93f14f66bcd23e542259 that the bootstrap of a
    distributed cloud system controller will fail due to the RVMC image
    is a string rather than a list. This commit converts it to a single
    member list before appending it to static images list.
    
    Tested by bootstraping a distributed cloud system controller.
    
    Change-Id: Ib2e0c6cc1913e0ee44f8b288004459c497ebedb8
    Closes-Bug: 1908100
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 6cf33738447ba9f65d0f5aec1507991676522a58
Author: Sabeel Ansari <Sabeel.Ansari@windriver.com>
Date:   Tue Dec 22 12:22:04 2020 -0500

Create deployment namespace during bootstrap
    
    A generic namespace is being created during bootstrap
    called 'deployment'. This namespace will be shared by
    various deployed platform resources including platform
    certificates.
    
    Change-Id: I48bca00cc882fae691786fd1e200e28bf126496b
    Story: 2007361
    Task: 41495
    Signed-off-by: Sabeel Ansari <Sabeel.Ansari@windriver.com>

commit 587c1ff2075c518f32bb93f14f66bcd23e542259
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Dec 14 11:55:32 2020 -0500

Upgrade: append additional images to the static images list
    
    Images specified in additional_local_registry_images at install time
    will not be upgraded after an upgrade is completed. This commit allows
    common/load-images-information load additional images list from
    /usr/share/additional-system-images.yml by default.
    
    In a distributed cloud system, the Redfish Virtual Media
    Controller(RVMC) image is can support remote install on Redfish
    configured hosts. This commit includes the RMVC image in the static
    images list if the host is a DC controller, enables
    download/push/update this image with other static images.
    
    Tested by installing and upgrading an AIODX central cloud with an
    AIOSX subcloud DC system.
    
    Partial-Bug: 1908100
    Change-Id: I1f927f876f4883a587098c61fbcaf408d65fdde4
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit a83a13a0e8c6923ea4e9f74797269aa9ee4bf02d
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Jan 11 00:46:54 2021 -0500

Ensure docker proxies are defined
    
    System triggered playbooks such as upgrade-static-images,
    will fail to pull newer images from AWS ECR if either
    docker http or https proxy is not configured. This commit
    resolves the issue.
    
    Tests:
      - Fresh install with public registries
      - Fresh install with a private registry
      - Fresh install with AWS registry, http proxy configured
      - Fresh install with AWS registry, proxies not configured
          a. Perform controller swact
          b. Perform upgrade
    
    Closes-Bug: 1910951
    Change-Id: I93c2856210dde90a66bae823178e14f140f711f4
    Signed-off by: Tee Ngo <tee.ngo@windriver.com>

commit bbcec5f2d678a0f18c02eb514e274a6187c21e32
Author: Angie Wang <angie.wang@windriver.com>
Date:   Wed Oct 14 15:56:39 2020 -0400

Configure SQL as helm storage backend
    
    Configmap is the default helmv2 storage backend to store
    release information but its 1MB resource limit prevents
    scaling up stx openstack worker nodes, so we want to use
    SQL as helm storage backend.
    
    Update armada overrides to start up tiller with SQL storage
    backend.
    
    Tested:
    - AIO-SX, AIO-DX, STD installation (IPv4 and IPv6)
    - apply stx-openstack
    - host-swact/lock/unlock
    - AIO-SX, AIO-DX upgrade with stx-openstack, stx-monitor
    - backup&restore with stx-openstack
    
    Closes-Bug: 1887677
    Depends-on: https://review.opendev.org/#/c/761647/
    Change-Id: I8ad7194973e8fc60ee2539dea5da67b15be1df4a
    Signed-off-by: Angie Wang <angie.wang@windriver.com>

commit 56e338e5c729242dd14a7ecd4d294d65ca069863
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Dec 16 23:52:26 2020 -0500

Improve bootstrap config validation
    
    A misconfigured docker_registries when merged with the
    default docker_registries will lead to bootstrap failure
    with a misleading error that is difficult for the user to
    identify the root cause.
    
    Add tasks to ensure user provided registry keys are valid.
    
    Closes-Bug: 1908479
    Change-Id: I829496481708221a772a9242dc660a2462880116
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 820f347324d4fb7d17396d5d21f98eb2b674d23a
Author: Zhipeng Liu <zhipengs.liu@intel.com>
Date:   Sat Oct 31 00:32:03 2020 +0800

Enable etcd with security setting.
    
    Add etcd client/server certificate generation process in
    ansible playbook and restart etcd after etcd security config
    enabled.
    
    Test status.
    1) Deployment test on both simplex and duplex  - PASS
       check the communication status between apiserver
       and etcd by kubectl command
       check the etcd status and configuration on controllers
    2) Switch active controller - PASS
       After switching, check the communication status between
       apiserver and etcd by kubectl command
       check the etcd status and configuration on controllers
    3) Lock/unlock of a simplex controller - PASS
    4) Backup/Restore test on simplex - PASS
    5) Spontaneous reboot of a simplex controller - PASS
    6) Enable secured etcd on simplex by script after deploy
       unsecured etcd. - PASS
    7) Backup test on Duplex - PASS
    8) Restore test on Duplex - PASS
    9) Re-installing a controller host on a duplex setup and
       then swacting to it - PASS
    
    Partial-Bug: 1894870
    
    Depends-on: https://review.opendev.org/#/c/760508/
    Change-Id: I88691b84c9acc2e27f0b783d7454a873d3490072
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

commit 0df607a49b514f143a9198c6fbba1e9e846e6c4f
Author: Jim Gauld <james.gauld@windriver.com>
Date:   Tue Jul 14 14:21:14 2020 -0400

Support upgrade to Helm v3 with containerized armada
    
    This provides an upgrades playbook for containerized armada
    to launch armada using Helm v3.
    
    This refactors common code from bringup_helm.yml so that it may be
    called from either bootstrap or upgrade.
    
    Story: 2007927
    Task: 40354
    Closes-Bug: 1906554
    
    Change-Id: I061006683252a8592f07c90d6d82bdc109418451
    Signed-off-by: Jim Gauld <james.gauld@windriver.com>
    Signed-off-by: Angie Wang <angie.wang@windriver.com>

commit 936881f954f8acb2ed0b69de47f576d49ed82d2d
Author: Cole Walker <cole.walker@windriver.com>
Date:   Mon Nov 30 17:27:56 2020 -0500

Check local pods only to prevent B&R timeout
    
    This fix reworks a kubectl wait command to only wait for ready pods on
    the local controller. This fixes an issue where restoring a cluster with
    many nodes can timeout during "Start wait for armada,
    calico-kube-controllers & coredns deployments to reach Available state".
    This timeout was happening because the kubectl command was waiting for
    pods on downed nodes to become ready for 30 seconds per pod. In cases
    where there were 5 or more unreachable pods, the async timeout value of
    120 seconds would be reached and the kubectl wait commands would be
    terminated before completion. This prevented subsequent Ansible tasks
    from completing and resulted in a parsing error during "Fail if any of
    the Kubernetes component, Networking or Armada pods are not ready by
    this time"
    
    The fix here adds --field-selector spec.nodeName=$(hostname) to the
    kubectl wait command and causes only pods on the running
    controller to be checked. This also ensures that the asyncronous tasks
    will always only take 30 seconds, regardless of the number of nodes in
    the cluser.
    
    Removed the behaviour where the wait time would scale up based on the
    number of nodes, as the return time of the async tasks is now always a
    fixed 30 seconds.
    
    Closes-Bug: 1905788
    
    Change-Id: Idb7b30891d4fd00901aaa69412ef4c59913e21f3
    Signed-off-by: Cole Walker <cole.walker@windriver.com>

commit 843e77819228d1298db192d68746f2591ac3e078
Author: Andy Ning <andy.ning@windriver.com>
Date:   Fri Dec 4 10:14:25 2020 -0500

Restrict permissions on docker registry certificate file
    
    It is noticed that docker registry certificate file
    (/etc/docker/certs.d/registry.local:9001/registry-cert.crt) has
    permission set to 644. This update changes its permissions to
    400 as required, by preserving the original permissions when it
    is copied over in ansible bootstrap.
    
    Change-Id: Ic85fa5fa2595f81d5cde6b0294eb7fbbd9c7dc63
    Closes-Bug: 1906844
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit d3e48692a5b28a2466ea9dbf8d0737bebd9df68e
Author: Cole Walker <cole.walker@windriver.com>
Date:   Tue Dec 1 16:50:41 2020 -0500

Lower negative cache TTL for coredns
    
    This change updates the coredns config map to lower the TTL for caching
    negative responses from 30 seconds down to 5 seconds. This will improve
    the response time for cases where a hostname lookup might occur before a
    given pod is created, resulting in a negative entry being cached and
    preventing pods from resolving the name for 30 seconds afterwards.
    
    The default cache size of 9984 items is unchanged, but must be
    explicitly defined in this configuration.
    
    The change also adds coredns to the upgrade-k8s-networking playbook to
    ensure that changes to coredns can be automatically deployed by
    sysinv-conductor.
    
    Closes-Bug: 1906870
    
    Change-Id: I7b44358508c32c8ca4e58b1e69d6232f1a61bfcf
    Signed-off-by: Cole Walker <cole.walker@windriver.com>

commit 1d7305eaf980ea258bcf9e2bb31406bf96ceb766
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed Nov 18 01:11:33 2020 -0500

Remove the addition of identity to shared services
    
    Identity services are no longer a shared service in DC.
    This commit removes the addition of identity to the shared service
    list for subclouds.
    
    Change-Id: I3d9d0e4df1a41142cce1ce13d4bbf7d43a626909
    Partial-Bug: 1904675
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

tags:

added: in-f-centos8

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-15: Change abandoned on distcloud (f/centos8)

#20

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/793405

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-15: Fix proposed to distcloud (f/centos8)

#21

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/796528

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-16: Fix merged to distcloud (f/centos8)

#22

Download full text (105.0 KiB)

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/796528
Committed: https://opendev.org/starlingx/distcloud/commit/4c5344f8765b372cb84d2b1181589c16db2ae6e4
Submitter: "Zuul (22348)"
Branch: f/centos8

commit cb979811017bd193fc1f06e53bb7830fd3184859
Author: Yuxing Jiang <email address hidden>
Date: Wed Jun 9 11:11:27 2021 -0400

Format the IP addresses in payload before adding a subcloud

    The IPv6 addresses can be represented in multiple formats. As IP
    addresses are stored as text in database, ansible inventory and
    overrides, this commit converts the IP addresses in payload to
    standard text format of IPv6 address during adding a new subcloud.

    Tested with installing and bootstrapping a new subcloud(RVMC
    configured) with the correct IPv6 address values, but with
    unrecommended upper case letters and '0'. The addresses are
    converted to standard format in database, ansible inventory and
    overrides files.

    Partial-Bug: 1931459
    Signed-off-by: Yuxing Jiang <email address hidden>
    Change-Id: I6c26e749941f1ea2597f91886ad8f7da64521f0d

commit 2cf5d6d5cef0808c354f7575336aec34253993b3
Author: albailey <email address hidden>
Date: Thu May 20 14:19:24 2021 -0500

Delete existing vim strategy from subcloud during patch orch

    When dcmanager creates a patch strategy, if a subcloud has an
    existing vim patch strategy, it will attempt to re-use
    that strategy during its patching phase, which may result in an
    error.

    This commit deletes the existing vim patch strategy in
    a subcloud, if it exists, so it can be re-created.
    If the strategy cannot be deleted, orchestration fails.

    Change-Id: Id35ef26ed3ddae6d71874fc6bac11df147f72323
    Closes-Bug: 1929221
    Signed-off-by: albailey <email address hidden>

commit 9e14c83f0162549a2a94cb8bc1e73dbc4f4d4887
Author: albailey <email address hidden>
Date: Tue Jun 1 14:37:14 2021 -0500

Adding activation retry to upgrade orchestration

    When performing an activation, the keystone endpoints may not
    be accessible in the subcloud due to the asyncronous way that
    cert-mon can trigger a restart of keystone.

    This would have occasionally resulted in the upgrade activation
    failing to be initiated, and orchestration needing to be invoked
    again to resume.

This 'hack' adds retries and sleeps to the initial
activation action.

    Change-Id: Ic757521dec7bdc248a51a70b5463caafe7927360
    Partial-Bug: 1927550
    Signed-off-by: albailey <email address hidden>

commit bb604c0a9b872efd65fa45f1e2269995818c6262
Author: Tee Ngo <email address hidden>
Date: Thu May 27 22:17:16 2021 -0400

Fix subcloud show --detail command related issues

    If the subcloud is offline, the command stalls and eventually returns
    the "ERROR (app)" output. If the subcloud is online, the oam_floating_ip
    info is excluded from the output when the subcloud id instead of subcloud
    name is specified.

This commit fixes both of the above issues.

    Closes-Bug: 1929893
    Change-Id: I995591368564539b0e6af185b1adba2db73e0e46
    Sign...

Reviewed:  https://review.opendev.org/c/starlingx/distcloud/+/796528
Committed: https://opendev.org/starlingx/distcloud/commit/4c5344f8765b372cb84d2b1181589c16db2ae6e4
Submitter: "Zuul (22348)"
Branch:    f/centos8

commit cb979811017bd193fc1f06e53bb7830fd3184859
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Wed Jun 9 11:11:27 2021 -0400

Format the IP addresses in payload before adding a subcloud
    
    The IPv6 addresses can be represented in multiple formats. As IP
    addresses are stored as text in database, ansible inventory and
    overrides, this commit converts the IP addresses in payload to
    standard text format of IPv6 address during adding a new subcloud.
    
    Tested with installing and bootstrapping  a new subcloud(RVMC
    configured) with the correct IPv6 address values, but with
    unrecommended upper case letters and '0'. The addresses are
    converted to standard format in database, ansible inventory and
    overrides files.
    
    Partial-Bug: 1931459
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: I6c26e749941f1ea2597f91886ad8f7da64521f0d

commit 2cf5d6d5cef0808c354f7575336aec34253993b3
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 20 14:19:24 2021 -0500

Delete existing vim strategy from subcloud during patch orch
    
    When dcmanager creates a patch strategy, if a subcloud has an
    existing vim patch strategy, it will attempt to re-use
    that strategy during its patching phase, which may result in an
    error.
    
    This commit deletes the existing vim patch strategy in
    a subcloud, if it exists, so it can be re-created.
    If the strategy cannot be deleted, orchestration fails.
    
    Change-Id: Id35ef26ed3ddae6d71874fc6bac11df147f72323
    Closes-Bug: 1929221
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 9e14c83f0162549a2a94cb8bc1e73dbc4f4d4887
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jun 1 14:37:14 2021 -0500

Adding activation retry to upgrade orchestration
    
    When performing an activation, the keystone endpoints may not
    be accessible in the subcloud due to the asyncronous way that
    cert-mon can trigger a restart of keystone.
    
    This would have occasionally resulted in the upgrade activation
    failing to be initiated, and orchestration needing to be invoked
    again to resume.
    
    This 'hack' adds retries and sleeps to the initial
    activation action.
    
    Change-Id: Ic757521dec7bdc248a51a70b5463caafe7927360
    Partial-Bug: 1927550
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit bb604c0a9b872efd65fa45f1e2269995818c6262
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu May 27 22:17:16 2021 -0400

Fix subcloud show --detail command related issues
    
    If the subcloud is offline, the command stalls and eventually returns
    the "ERROR (app)" output. If the subcloud is online, the oam_floating_ip
    info is excluded from the output when the subcloud id instead of subcloud
    name is specified.
    
    This commit fixes both of the above issues.
    
    Closes-Bug: 1929893
    Change-Id: I995591368564539b0e6af185b1adba2db73e0e46
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit f6a7ef1262d7e62e98cbcf1cefa45ce635b04e20
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Apr 22 09:01:41 2021 -0500

Cleanup Distributed Cloud Orchestration unit tests
    
    We have a DB setup that we can populate, so the subcloud DB values
    should not be mocked, and instead we should verify they are being
    properly set.
    
    Added unit tests for these states that had no tests:
     - kube upgrade 'updating patches'
     - kube upgrade 'deleting vim strategy'
     - kube upgrade 'upgrade simplex'
    
    Relocate a unit test that was in an old directory and was not
    being executed:
     - upgrade 'deleting load'
    
    Some tests were incorrectly changing the name of an object rather
    than its state attribute.
    
    hacking was incorrectly being specified directly within tox.ini
    rather than the test-requirements.txt file.
    
    FakeSubcloud is no longer being used anywhere.
    
    Story: 2008137
    Task: 42409
    Change-Id: I9d43a2b0f1923cbd7d469965968062503a58234d
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit d7986567d0111adeb175a9c106ad1f72b2e0c052
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu May 20 13:43:33 2021 -0400

DC dcorch log cleanup
    
    In order to increase retention period of dcorch logs, some info
    logs are moved to debug level.
    
    Change-Id: I8091b7fea6f0f8c3834802d03c438252529137ae
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1929111

commit 9f5b85c41c2d8aec2e550f187a8b40a7525033b0
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed May 19 14:12:07 2021 -0500

Specify the nodeset for zuul jobs
    
    The py2.7 jobs need to specify xenial
    The py3.6 jobs need to specify bionic
    The focal zuul nodes only have python 3.8 installed in them
    
    Zuul targets that invoke a generic python3 interpreter such
    as pep8 is not specified.
    
    The copyright date was updated in order to trigger
    the zuul jobs, as a no-delta type of change.
    
    Change-Id: If5352228a018f3147ae78172ae625066150c0a95
    Partial-Bug: 1928978
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 6b87e180298b19f5cfc441620cfdc546696ffb58
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed May 19 10:02:25 2021 -0400

Add support for k8s-upgrade-completed event notification
    
    This will be used by sysinv to notify dcmanager that the kubernetes
    upgrade has been completed so that it can trigger a kubernetes
    audit of all subclouds.
    
    Change-Id: I2ff48c939e808c3576a5739fa1dc37906e6a3a4e
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1928864
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792043

commit 077ab926e388d477a788e3cf306e2b7443c3927e
Author: albailey <Al.Bailey@windriver.com>
Date:   Fri May 14 13:48:23 2021 -0500

Updating some of the dcmanager and audit logs
    
    On startup of any process, the entire configuration was
    being logged, which was we were logging the entire
    configuration to the file (almost 200 lines per process).
    - Lowering that log to DEBUG.
    - Added an extra 'Starting...' log entry to help indicate
    when a process start/restart occurs.
    
    The update_subcloud_endpoint_status  log was not including
    enough information, as it is called several times in a row
    for each endpoint.
    
    Two audit logs that are created every 30 seconds have been reduced
    from info to debug.
    
    Closes-Bug: 1928335
    Change-Id: I7ada5abf87c2f28f5826c02345f8dd3197eae665
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 84dc98f5df97a8dba050a96e4cedace0e9419d8f
Author: Melissa Wang <melissa.wang@windriver.com>
Date:   Wed May 5 08:51:22 2021 -0400

Fix for subcloud upgrade activation failure
    
    This change adds exception-handling in case the haproxy is reset
    during upgrade activation so that it will retry the activation step
    
    Closes-Bug:1927550
    
    Change-Id: I678d50df3d8866518356d8da95b4742f0072c7fe
    Signed-off-by: Melissa Wang <melissa.wang@windriver.com>

commit 8d16d245504039b7e59b5d7a083b3b9d28b358e9
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 13 08:36:52 2021 -0500

Reset the force patch audit flag after it is triggered
    
    The force patch audit flag could be set, but never cleared.
    This would result in patch audits being run during every audit
    loop, as well as any derived audits such as software load.
    
    This change ensures that the force flag is cleared when a
    patch audit is invoked.
    
    The force patch audit flag is used by other audits such as the
    audit_load, so it needs to be reset after those evaluations.
    
    Change-Id: I833d157a616916a9ee37553b1f4cf3582caa8862
    Closes-Bug: 1928333
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 06d48e2ede3991d623089455e513918e07cf7ba4
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Thu May 6 13:51:32 2021 -0400

Extend the ansible timeout of the rehome playbook
    
    On a sub cloud that can no longer talk to its system controller,
    ansible cannot escalate the root privilege in the sub cloud until
    connecting to the LDAP server times out. This commit overrides the
    ansible default timeout(10s) to 180s of the re-home playbook, to
    prevent the re-home playbook failed by privilege escalation timeout
    before correcting the LDAP server address in a sub cloud.
    
    Tested:
    Re-homed a subcloud after powering off the previous system controller
    and deleting the static route to the system controller.
    
    Story: 2008774
    Task: 42408
    
    Change-Id: I65f9e0b7e242ef53a6fadcbab801ff39778b3d8d
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit b57f5ef65bea69db8563683c13d2299cccc35845
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed May 5 10:50:58 2021 -0400

Re-initialize RegionOne keystone client properly after subcloud add
    
    When a new subcloud is added, we expect IndexErrors while trying to
    fetch the subclouds identity auth URL. At this time, the RegionOne
    keystone client is also refreshed (by setting it to None). The
    next subcloud that tries to access the master keystone client
    cannot properly handle this and its availability is set to None for
    a short duration.
    This commit fixes the issue described above.
    
    Change-Id: I3139108b119aa6b6ba20277d829c14cc61e9f88c
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1927229

commit 17b5505d9ea9b149cf28236be3c1b4c263a89ffb
Author: Tao Liu <tao.liu@windriver.com>
Date:   Mon May 3 12:32:53 2021 -0400

Fix Sub clouds going offline due to auth failure
    
    This update contains the following changes that prevent subclouds
    going offline due to authentication failure:
    1. The os region client cache is cleared when a new keystone client
    is created. The os region client will be re-created using the new
    keystone session.
    2. When the user's access info (such as role id) is changed create
    new keystone client and os region clients. This could happen after
    system controller keystone role ids were synced to subclouds
    3. Remove get_admin_backup_session that was only required when
    upgrading to stx 4.0.
    4. Increase AVAIL_FAIL_COUNT_TO_ALARM to 2 as we don't want to alarm
    first failure since there are cases where we expect a transient
    failure in the subcloud (e.g. haproxy process restart to update
    certificates)
    
    Tested on DC-6:
    1. Adding 50 subclouds twice
    2. Soaking the fix over the weekend
    
    Closes-Bug: 1927007
    
    Signed-off-by: Tao Liu <tao.liu@windriver.com>
    Change-Id: I86fdc9a2f062409e704bdfac2119dc488123f7de

commit 8acd9699acec23c6ce4fe97bb1ea790d38032951
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed Apr 28 16:36:08 2021 -0400

Fix: Subcloud shows (patching) out-of-sync on the system controller
    
    The patching logic is modified to mark the subcloud as in-sync
    when the patches that are applied on the system controller
    have been committed on a subcloud.
    
    Change-Id: Ic56f0490cc0f7728ace721a2436d602a03ca1328
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1926635

commit 70a3475c73c851c4179d809d8e1b28ba46508683
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Apr 19 17:18:18 2021 -0400

Handling retry attempts of duplex subcloud upgrade strategy
    
    This commit supports retry attempts for duplex subcloud upgrade
    strategy when upgrade states are "data-migration",
    "data-migration-complete" or "data-migration-failed".
    
    Change-Id: I919228126a4960f712092c26b3abbf12e2de8ccb
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Story: 2008055
    Task: 42304

commit f5890513348e889824b61fda8867b14c7d0c4cde
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Apr 19 12:22:22 2021 -0400

Handle upgrade failure following unlock timeout
    
    In this commit:
    - Set the subcloud deploy status to DEPLOY_STATE_DATA_MIGRATION_FAILED
      if the subcloud host unlock/reboot times out so that the subcloud
      can be audited and handled correctly in pre_check upon upgrade
      retry.
    
    - Update pre_check to advance to activating step if deploy
      status is DEPLOY_STATE_DATA_MIGRATION_FAILED but the subcloud
      is online and passes health check.
    
    - Separate simplex and duplex pre_check logics for readability.
    
    - Correct a dcorch log that was not updated when the platform admin
      userid was changed from wrsroot to sysadmin.
    
    Closes-Bugs: 1925053
    Change-Id: Ie7f23a4f8de4e5dbcacfd7d625e15fe25d30e499
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit a9157c51d5562eb395815d4653d8e7d39b38920e
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Fri Apr 16 11:00:26 2021 -0400

Filter out skippable alarms in precheck
    
    Subcloud online checks are now skipped if the subcloud is
    already in the 'migrated' state. The precheck also skips
    upgrade alarm as well as the host lock alarm if upgrade
    has started.
    
    In addition, the commit includes the code to handle bmc_password
    with None value which the previous commit
    (69a74499884a8a73a3b705ce074f46202c4aa278) did not handle.
    
    Closes-Bug: 1924774
    Change-Id: Ifdfe1e44a34c2c2561c299a82d7178dce6063daf
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit 6c986880b7caa6687ffceaa1f3888d989756eb3e
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Apr 14 10:03:30 2021 -0500

Kube audit should report out-of-sync if a kube upgrade exists
    
    A subcloud can have all its kubernetes components upgraded, but
    there can still be a kubernetes upgrade object.  Audit will now report
    the existance of that object as meaning the subcloud is out of sync.
    
    Similar logic already exists for regular upgrades.
    
    Change-Id: If84c24bdd213c18fefde3752b1798f67aad560ae
    Story: 2008137
    Task: 42265
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit ef0123d0b48136edcef794ec03cad776100603e3
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Tue Apr 13 21:25:29 2021 -0400

Add restore related deploy states to auditable list
    
    In this commit, relevant restore related deploy states are
    added to auditable list so the subcloud can be set as offline
    while undergoing remote restore.
    
    Closes-Bug: 1923688
    Change-Id: Ib415327478c5ab45b61726f3ce2d6e88be0654cd
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit de96cf465b65661a60a8070f4293348fb962055d
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Apr 12 16:33:32 2021 -0400

Restore upgrade retry of offline simplex subcloud
    
    Code changes to support upgrade orchestration for duplex subclouds
    broke the upgrade retry functionality for offline simplex
    subclouds. This commit fixes it.
    
    Closes-Bug: 1923498
    Change-Id: I92d31df7ae8396293b302d357655470b447d143c
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 119dbaf87bb13a37e62b32d14dbd00776a1919db
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Feb 1 13:45:17 2021 -0500

Add a migrate option for rehoming a subcloud
    
    This commit adds a "migrate" option in dcmanager subcloud add API.
    With this option, the dcmanager can invoke a rehome_subcloud playbook
    to migrate a subcloud under another central cloud. The playbook will
    be installed under:
    /usr/share/ansible/stx-ansible/playbooks/rehome_subcloud.yml. The
    subcloud info will store in dcmanager database.
    
    The keystone password of "smapi" is required during the subcloud
    migration, so it is requested from keyring and added into the
    subcloud's overrides file.
    
    Tested:
    Happy path:
    On a system controller, trigger a successful run of the
    rehome_subcloud playbook with command:
    dcmanager subcloud add --migrate --bootstrap-address
    <bootstrap-address> --bootstrap-values bootstrap-values.yml
    In the boostrap-values.yml the systemcontroller_gateway_address should
    be set under the management subnet.
    The subcloud info is added into the dcmanager database. And its
    deploy_status is changed in the sequence: pre-rehome-> rehoming->
    complete
    
    Unhappy path:
    1 On a system controller, trigger a failed run of the rehome_subcloud
    playbook with the correct bootstrap values. The subcloud info is also
    stored in the dcmanager database. But the deploy_status is changed
    into rehome-failed in the end.
    2 The systemcontroller_gateway_address is not under the management
    subnet. The rehome_subcloud playbook is not triggered, and a notice
    indicates the problem is prompted.
    3 Create an error during creating the intermediate ca cert in
    cert-manager, subcloud's deploy status changes to "rehome-prep-failed"
    in the end.
    
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/780101
    Story: 2008774
    Task: 42150
    Change-Id: I758e102d386a63c7ed318bcc91d66e11d3f56299
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 69a74499884a8a73a3b705ce074f46202c4aa278
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Apr 8 14:40:30 2021 -0400

Encode bmc_password received from the subcloud host
    
    If the subcloud is configured to store bmc secret in its
    barbican database, ensure the password is base64 encoded
    before passing the install data to rvmc.
    
    Closes-Bug: 1923073
    Change-Id: I9639c8e09acb2d78013932d4123511b354bb04fb
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 8dcb56bb493078f34fec3c4f07786112cfbfacdb
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Mar 19 17:18:29 2021 -0400

Reduce keystone load for offline subclouds
    
    When a large number of subclouds are offline (e.g. powered down),
    the dcmanager-audit generates significant load on the keystone
    services in the System Controller when it periodically (roughly
    once per minute) creates a keystone token and then retrieves the
    endpoints. This causes CPU usage to spike. In order to address
    the issue, RegionOne client is cached in EndpointCache.
    
    Change-Id: I17db5cd0107457f07fe14febfec23da5290153de
    Story: 2007267
    Task: 42082
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 4a49932841abd3ca4f70eca500b37e60ba3ed746
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon Apr 5 15:01:53 2021 -0500

Only install yamllint in py3 env
    
    A new yamllint was released which dropped py2 support.
    Zuul servers that have updated their caches will attempt
    to install that version, as it is not clamped by upper constraints.
    
    This also attempts to not use site packages.
    
    Partial-Bug: 1922590
    
    Change-Id: I8b2776f4c12447a66743a8b2626aa914969dcd2a
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit b0b3cfe07b591648a12b669e36e6cc218d5328c3
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Apr 1 15:38:30 2021 -0400

Add API retries in finishing firmware update
    
    In firmware update orchestration, after the subcloud is unlocked,
    the runtime manifests on the subcloud may take some time to complete
    causing the sysinv API calls to query the subcloud to fail.
    This update is to add retry attempts during the finishing firmware
    update strategy step.
    
    Closes-Bug: 1922281
    
    Change-Id: I678eaf701d9c0cf40907fb8c378a5a2c531a88ae
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit d5fbf8913f0990f5d0e3b99edb79a8d8da801b12
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Mar 12 13:43:49 2021 -0500

Objectify firmware audit data
    
    Recently merged commit broke firmware audit and caused
    firmware_sync_status to be unknown even after the
    device image was uploaded on system controller. This
    commit fixes firmware audit by objectify the firmware
    audit data similar to patch and kubernetes audit data.
    
    Change-Id: Ibb53641b9ead78767523612a1cff8c8993c874dd
    Story: 2007267
    Task: 42048
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 36497a9bf518c52338e5f46d2f541124d742a8c2
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Mar 10 09:05:19 2021 -0500

Add unit tests for subcloud restore
    
    Update API and manager unit tests to include subcloud restore
    tests.
    
    Story: 2008573
    Task: 42028
    Change-Id: I0e16f9914fb5a4d0b73ea9837a75931e2a4432d6
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 7625b448de19d0a66ed3c98793e254671404e801
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Mar 8 16:51:09 2021 -0500

Handle in-progress subcloud operations after dcmanager-manager restarts
    
    When a subcloud is added it can go through several phases - install,
    bootstrap and deploy. If the dcmanager-manager process is restarted
    while the subcloud is being added, the DB would never be updated
    and subsequent phases are not completed. This commit fixes this
    issue by making dcmanager audit for subclouds in transitory states
    on startup and update the deploy state to the appropriate failure
    state.
    
    Story: 2007267
    Task: 42026
    Change-Id: I90a57cef89f1cc04a07fa6482ff2c29edf7be965
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 1348af4772414959561e40faa5d5a97e3cea11ce
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Mar 10 13:28:13 2021 -0500

Update dcmanager API doc for subcloud restore
    
    Document the dcmanager subcloud restore API.
    
    Story: 2008573
    Task: 42034
    Change-Id: I38505518b367b275a95f040cee67e6a183cf9606
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit c4a208325c6dd8403c9a93b74c50f1e7801097b0
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Wed Mar 10 10:19:51 2021 -0600

Remove existing dc-vault load data during import
    
    During a distributed cloud upgrade, an admin may need to delete and
    import a new load. If the load-delete command is issued without the
    --os-region-name SystemController parameter the load will be deleted
    from sysinv but will still be present in the dc-vault.
    
    To resolve this we will always remove any existing dc-vault data during
    a load import.
    
    Closes-Bug: 1918451
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>
    Change-Id: I143e2c084205c105f084d96714e8ebef94dbd130

commit 8e26a6731ae97b05f399dc52709b50eaa59054b8
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Mar 2 13:25:14 2021 -0600

add support for event notifications to dcmanager
    
    Add support for generic event notifications to dcmanager.  Initially
    this will be used by sysinv to notify dcmanager that the platform
    upgrade has been completed so that it can trigger a load audit of
    all subclouds
    
    Story: 2007267
    Task: 41968
    Change-Id: I9edb00a385ee5f617ac4415aa7e4675f52e28139
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>

commit b6638ce9a1055e5131a494d70897cbf37ce2a8c1
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Nov 17 15:47:58 2020 -0600

Distributed Cloud Kubernetes Upgrade Orchestration
    
    This commit introduces DC Kubernetes upgrade orchestration.
    
    - Added a new kubernetes endpoint for DC audit
      The audit compares that the active-target-version in the subcloud for
     kubernetes matches the system controller.
    - Wiring the kubernetes strategy to the kube orch thread
    - Add protections to orchestrator shutdown for when 'start' encounters
     a problem.
    - Added 'kubernetes' as a valid strategy type for:
       create / abort / delete / show
    - Update the endpoint flag file name to trigger updating the new
     endpoint
    - Added a kubernetes orch thread similar to sw upgrade orch thread
     with these states:
      - updating kube patches
      - creating vim patch strategy
      - applying vim patch strategy
      - deleting vim patch strategy
      - creating vim kube upgrade strategy
      - applying vim kube upgrade strategy
    
    Patch vim orchestration steps will be skipped if there are no patches
    that need to be applied in the subcloud.
    
    Change-Id: Ic508ed1a504dba177c100a9caf783e87bc10193a
    Story: 2008137
    Task: 41177
    Depends-On: https://review.opendev.org/c/starlingx/nfv/+/767421
    Signed-off-by: albailey <al.bailey@windriver.com>

commit 1602fc70b271c8986c0ba414300a9b27ff5f2e9d
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Fri Feb 26 08:57:11 2021 -0600

Support background runtime manifests during upgrade-activate
    
    Update dcmanager orchestrator to accept the new activating-hosts state.
    
    Story: 2008055
    Task: 41947
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>
    Change-Id: I032d82ed489b1ae0652429f91adfa5fa43fc3d3f

commit af14a6f65cfa13746ef9bed29a59e9640b4d1092
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Tue Feb 23 00:09:57 2021 -0500

DCManager update for Redfish subcloud restore
    
    In this commit, dcmanager api and manager are updated to
    support the restore of a Redfish capable subcloud from
    backup data.
    
    Unit tests will be added in a separate commit.
    
    Tests:
      - Restore request without install (not yet supported)
      - Restore request for a subcloud that is currently in 'managed'
        state
      - Load is not in dc-vault
      - Mandatory restore value (backup_filename) is not present
      - Specified backup file cannot be found
      - Subcloud restore with backup data on the target (i.e.
        backup tarball is under /opt/platform-backup)
      - Subcloud restore with backup data on the system controller
        (i.e. on_box_data = false)
      - Simulate install failure
      - Simulate check target failure
      - Batch subcloud restore
      - Subcloud restore retry
      - Subcloud restore with patches
    
    Task: 41725
    Story: 2008573
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/777046
    Change-Id: I8134c535e39231837727811475b0f01b2ccddb63
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 69b815af7d1e20eebc6fef2fe4b514a3355359ad
Author: Adriano Oliveira <adriano.oliveira@windriver.com>
Date:   Thu Feb 11 21:02:41 2021 -0500

Add orchestrator unlock retry logic
    
    During distributed cloud orchestrator upgrade, a SR-IOV VFs
    configuration change may take several minutes to apply because the
    runtime manifest may have delayed for several minutes.
    While not applied, if an unlock is attempted, it will fail because
    sysinv prevent it to happen since DB was not updated yet.
    Code has been added to detect host unlock failure and implement a
    retry logic that can take up to 20 minutes.
    
    Partial-Bug: 1914836
    Signed-off-by: Adriano Oliveira <adriano.oliveira@windriver.com>
    Change-Id: I51b1eba8cf7cba1368e159f9dbf0b4d6ee858285

commit c446fb2f2ff58dc3f4c4708ff5aae259d75ff4d1
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Feb 16 17:11:58 2021 -0600

dcmanager update availability on first failed audit
    
    In order to improve the response time in the face of failures,
    change the dcmanager aduit code to update the availability
    (offline/online) on the first failed audit rather than waiting
    for a second failure.
    
    This does mean that an intermittent failure will result in
    dcmanager treating the subcloud as "offline".
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: If71d51e82a0147ba38153bf06a6fa8f54d38ae80

commit 0a6ce1a956e6d9474c69a26d34ec733716b821b4
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Thu Feb 25 10:05:56 2021 -0600

Trigger all sub-audits when subcloud goes from offline to online
    
    When we detect a subcloud going from offline to online, we can't
    assume anything about it anymore so let's trigger all the sub-audits
    so that on the next audit we'll know whether it's in sync or not.
    
    (I also re-raised a couple of exceptions to get better error
    handling.  Shouldn't ever hit this but you never know...)
    
    Closes-Bug: 1916784
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ie252272ff75077e21a9942f136ebfbd95420c175

commit 3b2fb67cfe1556fb468363c5196056d54c8aad6b
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Feb 22 13:18:24 2021 -0600

trigger subcloud audit when subcloud becomes managed
    
    When we change a subcloud from unmanaged to managed, trigger all of
    the various subcloud audits for that subcloud.
    
    Previously we triggered them for *all* subclouds, but now we have
    the ability to do it for just one.
    
    Also change it to only trigger the audits if we successfully change
    the subcloud to "managed".  No point in triggering them if we
    weren't successful or if it was already managed.
    
    Story: 2007267
    Task: 41889
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: I4494c80080dacb59d330162035bd6b595d8026d2

commit 0b38bccc5f13ed4bf661d7b47a799c2eb2bbe1e2
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Feb 16 11:04:03 2021 -0600

add API to run all subcloud audits for one subcloud
    
    Add a SubcloudAuditManager API to trigger all of the subcloud audits
    for a specific subcloud.  This would be useful when adding a new
    subcloud.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: I015c7c5762a0fca1f08ff6e1aa92ca7a738ddb82

commit 843545184a89c1ae4c4907c7b8169a9eb60e29c7
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Fri Feb 12 12:23:48 2021 -0600

Trigger audits on dcmanager-audit startup
    
    If we swact controllers or patch dcmanager-audit, the audit processes
    can potentially be killed while there are audits in progress.
    
    Rather than wait for the full timeout, we trigger an "audit fixup"
    on dcmanager-audit startup, and also trigger all of the sub-audits
    in case we were doing any of them when the audit processes were
    killed.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ic069b6d10cae77613571faaba10869367182c118

commit fe05f2ea9bb6eba394229ad019b23f54eac31b69
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Jan 26 17:38:17 2021 -0600

fix up stale subcloud audits
    
    If something happens and an audit is interrupted after it was
    started but before it completed, the timestamps will be messed up
    in the DB and a new audit won't be triggered because the audit
    manager will think that there's an audit already in progress.
    
    Add a new check that looks for subclouds where the audit has been
    "in progress" for long enough that we can be pretty sure something
    has gone wrong.  Once we find them, reset the audit finish time
    so that a new audit will be triggered at the next audit start time.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: I390a99ba2bb06915b60fcfb8865096e1ede36cd3

commit c71703128e99bdd47532524716684908d2a734ff
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Nov 30 17:46:11 2020 -0600

Move subcloud audit to new worker processes
    
    In order to allow for faster subcloud audits, introduce a new
    "audit-worker" process.  By default there will be four worker
    processes to perform the actual subcloud audits.
    
    The main audit process will scan the DB for subclouds which need
    auditing based on the new audit start/end timestamps.  It will
    then send out RPC messages to the "audit-worker" processes to
    request an audit for a list of subcloud IDs.
    
    When the "audit-worker" process receives the "audit_subclouds" RPC
    message, it loops over each of the specified subclouds.  For each
    subcloud it updates the DB to indicate that the audit is starting
    for that subcloud, then basically does the exact same audit that
    is currently done in the audit process, then updates the DB to
    indicate that the audit has been completed for that subcloud.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ifb3dd363fd337d24f2c3f7aaa3549624fffaceca

commit 8417ce7737a9798c0938a6e2d378db2961cb4636
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Nov 30 17:38:07 2020 -0600

Add subcloud audit DB table
    
    In order to allow for subcloud audits to be handled by multiple
    processes, add a new DB table to track subcloud audit information
    including start and finish times for the various audits as well as
    whether a special audit has been requested.
    
    On DB upgrade we populate the new table with entries for each
    non-deleted subcloud.
    
    The subcloud creation code is modified to also create a corresponding
    subcloud_audit entry.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Id2f2b49eb5b8b1f20b1c128bce4811479bcb96b9

commit a6cb0594193ce9cec4934a5b1b588b58d7f36082
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Sep 28 17:57:47 2020 -0400

Upgrade Orchestration for Standard subclouds
    
    This commit enhances upgrade orchestration to support Standard
    subclouds.
    
    Strategy Steps:
    - Pre-checks to verify subcloud is ready for an upgrade
      and to recover from previous failures
    - Install release N+1 license in subcloud
    - Download/import the release N+1 load in subcloud
    - Update and apply N+1 patches to release N+1 load in
      subcloud
    - Finishing patch strategy
    - Start duplex upgrade
    - Lock controller-1
    - Upgrade and re-install controller-1
    - Unlock controller-1
    - Swact to controller-1
    - Create VIM upgrade strategy
    - Apply VIM upgrade strategy
    - Swact to controller-0
    - Activating Upgrade
    - Completing Upgrade
    - Delete release N load
    
    Change-Id: I6fb60912ec1e90baa593cf98e502f4c91f599f42
    Story: 2008055
    Task: 41105
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit aab68918c9f307330692fa2cbff26a558566f066
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Wed Dec 16 14:36:53 2020 -0600

cleanup audit interval constants
    
    This converts the hardcoded patch interval to make use of an
    already-defined but unused config option, and tweaks the default
    audit interval.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ie6f023d0e46accdb79bd854e9833158e0a33e4b3

commit 527a48513a2534ed73755f758e01992414cabe21
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Fri Jan 22 14:32:18 2021 -0600

Turn off the legacy pip resolver for distcloud
    
    The requirements that had conflicts have been updated.
    The local upper-constraints is no longer being used by tox
    since two constraints files with different values for the same
    requirements are not supported. To enable this change, some
    unnecessary code that required an older version of
    oslo.messaging is removed.
    
    Bugbear is also being disabled due to brokeness that can cause
    test failures.
    
    Change-Id: If007b7e0ad3c04b78b619542162267cc7d502460
    Story: 2004515
    Task: 41670
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 13092996ca4d3224354bd8dd39a9c50e256d5646
Author: John Kung <john.kung@windriver.com>
Date:   Thu Jan 14 09:20:02 2021 -0600

dcorch-engine endpoint type handling improvements
    
    Handle the endpoint type remove and add operations. This is
    invoked for optional applications such as stx-openstack.
    
    Cleanup unused SubCloudEngine class.
    
    Tests Performed with multiple subclouds:
      Bootstrap, Installation and Deployment
      Initial manage and sync
      Resource update and sync via proxy and audit
      Add and remove endpoint type
      Unmanage and manage subclouds
    
    Change-Id: I5c7033e7bed91a7bcc9b2c989daf333b76013b8c
    Story: 2007267
    Task: 41589
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 43138e6aa59bf057ae4c7d2cda623ed042fc2fc1
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Tue Jan 5 10:27:34 2021 -0500

Specified RVMC image tag for subcloud install
    
    In local docker registry, it is possible that there are multiple
    versions/tags of Redfish Virtual Media Controller(RVMC) image, caused
    by upgrade/downgrade platform for example. Currently, the subcloud
    RVMC install picks only the first returned image tag which may not
    compatible.
    
    This commit specifies the required RVMC image tag. And this image will
    be downloaded/pushed to the local docker registry as a static image
    after a fresh install or upgrade by the ansible playbook.
    
    Closes-Bug: 1908100
    Change-Id: Ic2a593ce81952e4b43f41ee9908b59ba51fe92ab
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 547b164fa3e72267c47c9ee344a75306d2f88460
Author: Tao Liu <tao.liu@windriver.com>
Date:   Mon Jan 4 10:25:53 2021 -0500

Orchestration Support for a Single Subcloud Group
    
    This update adds an option to create an orchestration strategy
    for a single subcloud group. It encompasses all types of DC
    orchestration - patch, upgrade, firmware.
    
    Test:
    Create patch-strategy with or without --group option
    Create upgrade-strategy with or without --group option
    Create fw-update-strategy with or without --group option
    
    Story: 2007267
    Task: 41524
    Signed-off-by: Tao Liu <tao.liu@windriver.com>
    Change-Id: Icda88d1bbfa537f3e9e399d7fabb48e2b2b6a7fa

commit ec8dba8a53b1beb0562bdd4228860f42e8d055e2
Author: Takamasa Takenaka <takamasa.takenaka@windriver.com>
Date:   Wed Dec 23 09:10:33 2020 -0300

Remove entries related to host-based snmp
    
    According to the host-based SNMP removal, remove community
    and trapdest related processes
    
    Story: 2008132
    Task: 41394
    Change-Id: Icad7f82d5e20149a49cbb1540486b882856ac189
    Depends-On: https://review.opendev.org/765381
    Signed-off-by: Takamasa Takenaka <takamasa.takenaka@windriver.com>

commit 181bd0c86d77dd783d0d1c82acdc0063732ed073
Author: John Kung <john.kung@windriver.com>
Date:   Tue Dec 8 10:43:50 2020 -0500

Enable multiple dcorch-engine worker processes
    
    Multiple worker processes are created for distributing the dcorch
    sync and audit work of the dcorch-engine on the active platform
    controller host.
    
    The database sync lock is introduced as a synchronization mechanism
    for dcorch-engine workers and stores subcloud, endpoint
    and action locks.
    
    The subcloud sync status table is introduced to track subcloud/endpoint
    sync request, audit status, and report status.  In-memory caching of
    subcloud objects and tracking info such as last audit and reported sync
    are removed and the dcorch DB is referenced instead.
    
    The GenericSyncManager and sync_threads are restructured and use
    thread pools for work to be performed on subclouds.  The infrastructure
    allows for sync_threads to be allocated to a dcorch-engine worker.
    In this commit, each worker process can handle each subcloud.
    
    Robustness audits ensures no stale sync_lock and dcorch-engine
    worker process file limits are increased.
    
    Tests Performed with multiple subclouds:
      Bootstrap, Installation and Deployment
      Initial manage and sync
      Resource update and sync via proxy and audit
      Unmanage and manage subclouds
      Verify sync_lock on controller swact and process termination
      Dead Office Recovery
    
    Co-Authored-By: Tao Liu <tao.liu@windriver.com>
    Co-Authored-By: Teresa Ho <teresa.ho@windriver.com>
    
    Change-Id: I179279dea5babaf57084b28adea86a03b330924a
    Story: 2007267
    Task: 41374
    Task: 38662
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 37959399c9d1e5a8915bf0df80ba51136a5e91f4
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Dec 9 12:13:20 2020 -0600

Fix bugs with assigning a group for a subcloud
    
    The subcloud CLI and API support passing a group by
    name or ID, however the code was only working if an ID was passed.
    
    Furthermore, it could bypass the validator and allow setting zero
    as the group id, even though there is no group with an ID of zero.
    
    This submission includes unit tests for updating the group by ID,
    name, and with a number of rejection tests.
    
    Change-Id: I53d20a090cfe031c55802104012cea1cb4ff6cb6
    Closes-Bug: 1907488
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 9f0c5426cc54775ec4f9d2f0ff89b709ae95b5fc
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Dec 8 16:07:51 2020 -0600

Fix tox failures from new resolver in pip
    
    New pip was released on Dec 3, 2020 which breaks downloads
    from pypi for some URLs.
    
    New virtualenv was released Dec 7 which includes the new pip.
    
    The new pip uses a different resolver which fails with many
    of the components installed by pip.  These components
    may need to be fixed by later commits, since the legacy
    resolver is being removed completely in early 2021.
    
    Change-Id: Ifcbd8965e2ef0b53dae0c878c2a0f2bd91b572c2
    Closes-Bug: 1907125
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 164e8be8dd972031547127b4c1a820dc27a4886c
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Thu Dec 3 10:12:49 2020 -0600

Fix pylint errors
    
    Fixing remaining pylint errors and removing from pylint disable list
    so they cannot be re-introduced in the future:
    E1102: not-callable
    E1120: no-value-for-parameter
    E1128: assignment-from-none
    
    Change-Id: I0f59fdcd0fd20e9d0a8042ba3f7f2a5a20a701db
    Story: 2007082
    Task: 41335
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit a5ad984ba2990a4cf42479c45bd4a246e4ac778e
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Nov 30 11:27:41 2020 -0500

Add a specific counter to handle activation failure
    
    We intend to retry activation 10 times if it fails(in a total of 45
    minutes) in the DC orchestration. As the audit and retry are sharing
    the same counter, if the activation fails after 10 minutes, the
    orchestrator will not re-attempt the activation.
    
    To handle the activation failure, this commit adds a specific counter
    to activate retry in case the activate fails after 10 minutes. After
    this commit, the orchestrator will wait up to 45 minutes if the
    activation is in progress, and the retries will be limited to either
    the audit reaches DEFAULT_MAX_QUERIES or itself reaches
    MAX_FAILED_RETRIES.
    
    Tested by activating an AIOSX subcloud by orchestration.
    
    Closes-Bug: 1904934
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: Ic37c14ee316e02f690b55b082862a85fc54501bc

commit 326b8e33ddfc5854704a49b930f05229faf2124b
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Mon Nov 30 13:30:50 2020 -0600

Fix pylint E1101 (no-member) errors
    
    Fixing pylint E1101 errors and removing from pylint disable list
    so they cannot be re-introduced in the future.
    
    Some of the fixes included deleting or commenting out code that
    was no longer in use and was causing these errors.
    
    Change-Id: I5c5a7f9e90cc5cb9cf4510f578bb6e31cea1f5ef
    Story: 2007082
    Task: 41335
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit f7728125e1284edb064d471a92388b7dea074b80
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Nov 19 17:08:41 2020 -0500

Fix a logic flaw in orchestration pre-check
    
    If the subcloud is online but its deploy status is install-failed
    and the subcloud host is locked, the upgrading simplex step must
    have failed early in the previous upgrade attempt. The pre-check
    should transition directly to upgrading simplex step in the retry.
    
    Closes-Bug: 1904938
    Change-Id: I73815939526310d04d7b50dc8b85d33b5d502d25
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit b466efdb7e7e9ae2e45350a8d4cbe322383cb47c
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Sep 24 17:51:02 2020 -0400

Disable ansible logging when invoked via subprocess
    
    In this commit:
      - Overwrite the default ansible log path in dcmanager.
        The output of ansible playbook executed by dcmanager
        is already directed to /var/log/dcmanager directory.
        The default ansible logging to the playbook user's
        home directory (/root in this case) is unnecessary
        and can exhaust rootfs over time.
      - Write playbook output to /var/log/dcmanager/ansible
        directory.
      - Set separate logrotate rules for ansible output.
    
    Tests:
      - deploy subclouds and verify that
        a) no ansible output is written to /root
        b) ansible output (remote install, bootstrap, deploy)
           is written to /var/log/dcmanager/ansible, one single
           log per subcloud
        c) each instance of ansible output in the log file
           follows the following format: playbook command with
           timestamp followed by the output of the executed
           playbook
        d) subcloud ansible output log is rotated with delayed
           compression to ensure logging continuity for the
           on-going ansible play
        e) rotation rules (size, count) are respected
      - orchestrate a subcloud upgrade and repeat the above
        verifications
    
    Closes-Bug: 1895607
    Change-Id: Idbf87e4472df26c844526328aeacd9e55325bf0c
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit e88f3ef6f504c8b72fb4a3f9650a7f71e8ae6fb9
Author: Lu Yao Chen <luyao.chen@windriver.com>
Date:   Mon Nov 9 15:03:23 2020 -0500

Flag secret to mask dcmanager logs
    
    Using oslo_config flag and setting secret=True on sensitive info
    tested on running controller-0 system, api, audit and engine
    logs are masked by redacting the info
    
    Partial-Bug: 1896116
    
    Change-Id: I201ef344902ace391f6fa028ce81fd3fe96b92f3
    Signed-off-by: Lu Yao Chen <luyao.chen@windriver.com>

commit 83155e5e9f94bc1ab2e137f9b025b64119283e6a
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Tue Nov 3 20:23:39 2020 -0600

Update unit tests to use database instead of mocking
    
    Updating the distributedcloud unit tests to use the database
    during unit testing instead mocking all database access.
    This improves code coverage and simplifies the unit tests.
    
    Also fixing a minor bug in SubcloudManager.add_subcloud that
    resulted in an incorrect deploy_status being returned to
    the caller of the method (uncovered by unit test changes).
    
    Change-Id: I68eb9b98d440ae0c5f6661b17a590e6d7b1ff414
    Story: 2007082
    Task: 41210
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 561b24b285bbe5d3e651b34bc4136bc206f42af3
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Thu Oct 29 11:33:31 2020 -0500

Remove distributedcloud dependency on sysinv
    
    The distributedcloud package uses a single constant from the
    sysinv package. Removing this dependency - use of sysinv
    should only be through the REST API or the client - no
    sysinv code should be imported.
    
    Also changing tox config to move the tox work directory
    out of the source tree.
    
    Change-Id: I9973f16ae66a7339d2df958b5f2afe047ee980d2
    Story: 2007267
    Task: 41176
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 12b81ff452ce79d6dc1547c0a315284309e95b7f
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Sep 1 08:02:57 2020 -0500

Delete subcloud VIM strategies when DC fpga strategy deleted
    
    As part of an fpga upgrade strategy in a subcloud by the VIM,
    that strategy needs to be deleted.  Previously it was being
    deleted as part of a the final steps of a successful orchestration,
    but there can be scenarios where that orchestration is incomplete,
    and therefore these VIM strategies need to be handled.
    
    A new DC orchestration cannot be created until the old one is deleted,
    so this change ensures fpga vim strategies are deleted if possible
    when the DC orchestration strategy is deleted.
    
    A bug in applying vim strategy has been fixed, and the section removed
    since a vim auth exception cannot be raised since a new vim client
    is constructed per loop.
    
    Unit tests to invoke the delete strategy have been added.
    
    Tox coverage now generates an html report.
    
    Change-Id: I4f3d7bc3550e12c430d1b1c0ce3dffb298fe31b4
    Closes-Bug: 1891521
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 797574962804762a562a088b08b114f929ed6512
Author: Don Penney <don.penney@windriver.com>
Date:   Tue Oct 13 09:45:52 2020 -0400

Setup distcloud logfile configuration
    
    This update creates and packages distcloud-specific logfile
    configuration for syslog and logrotate, moving it out of the central
    syslog-ng-config package.
    
    Change-Id: Ia6d38fe7d39ff68178441db8844c00232f7597f6
    Story: 2008251
    Task: 41099
    Depends-On: https://review.opendev.org/757947
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit c659a7f684ca06e8104a66e95047d61966c865c2
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Aug 17 17:42:08 2020 -0400

Create an API to reinstall a subcloud
    
    This commit creates a REST API to reinstall a subcloud.
    
    In SubcloudController, the tasks includes:
    1. Read the subcloud info from dcmanager db, subclouds table.
    2. JSONify the data_install from db, generate the new install values.
    
    In SubcloudManager:
    1. Check the subcloud availability, software version before reinstall
    2. Check the image value, if image doesn't exists, update the image with
    iso in dc-vault
    3. Run the install command and apply command
    
    API format: PATCH /v1.0/subclouds/<subcloud>/reinstall
    -----
    Tests:
    -----
    Happy path:
    1. dcmanager subcloud add: a new subcloud with bootstrap values,
    deploy config and install values.
    2. dcmanager subcloud delete: an existing subcloud.
    3. dcmanager subcloud reconfig: an existing subcloud with deploy
    config.
    4. dcmanager subcloud reinstall: an existing offline subcloud, this
    subcloud has image path in data_install in db dcmanager. After the
    tasks of reinstall and bootstrap succeed, reconfig this subcloud with
    proper deploy config, this subcloud will be online.
    5. upload an image to dc-vault using: system --os-region-name
    SystemController load-import -a <bootimage.iso> <bootimage.sig>. Then
    using dcmanager subcloud reinstall to an existing offline subcloud,
    this subcloud has no image path in data_install in db dcmanager. After
    the tasks of reinstall and bootstrap succeed, reconfig this subcloud
    with proper deploy config, this subcloud will be online.
    
    Unhappy path:
    1. dcmanager subcloud reinstall: an existing online subcloud,
    reinstall fails.
    2. dcmanager subcloud reinstall: an existing subcloud without
    data_install value in db, reinstall fails.
    3. dcmanager subcloud reinstall: an existing subcloud with
    data_install in db, but missing mandatory install value, reinstall
    fails.
    4. dcmanager subcloud reinstall: an existing subcloud, but its sw
    version in db doesn't meet the sw version with system controllers',
    reinstall fails.
    5. dcmanager subcloud reinstall: an existing subcloud with
    data_install in db, but has no image path in data_install, and also
    has no right versioned image in dc-vault in system controller,
    reinstall fails.
    
    Story: 2007267
    Task: 40732
    Change-Id: I8be6d8d11e6b4ee02bbcca499ba8869ba76bffaa
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 19790859f060c35b28e18b482a2406729c9a5915
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Sep 24 09:48:09 2020 -0400

Use cached keystone client in add subcloud workflow
    
    Leverage keystone client caching work done for dcorch to reduce
    load to keystone during batch subcloud deployment.
    
    Tests:
      - Deploy 1000 subclouds in 50-subcloud increments in a simulated
        DC environment.
    
    Story: 2007267
    Task: 40955
    Change-Id: I716b202005b943a1f389d577ce035371d7ab938a
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit eb97f4c8b6b3c1cce6c162c76b5309751cb7a924
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Aug 24 13:19:14 2020 -0400

Move dcmanager orchestration to a separate process
    
    1) Remove DC manager orchestration from dcmanager-manager process
    2) Create dcmanager-orchestrator process and associated files
    3) Add new RPC calls for dcmanager-orchestrator process to notify
    dcmanager
    4) Create/update unit tests, to verify the implementation
    changes
    
    Story: 2007267
    Task: 40734
    Change-Id: Ibbbae77558a8a8fd95b636fa6c3aebb1dfefb514
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 64caf6de7c0ba23fa0f5c92facc19400a11a3c4e
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Mon Aug 24 14:12:18 2020 -0400

Import load metadata for SX upgrades
    
    For SX upgrades use the load-import-metadata API to import the load.
    
    Do not sync patches for loads that were imported with metadata only.
    
    Change-Id: If106711c976c1514419b120ba933662014a9bd7a
    Story: 2007403
    Task: 40807
    Depends-on: https://review.opendev.org/#/c/749413/
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>

commit 4c29aa748cc1d4b1540cbbfe8096d35855119a65
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Wed Aug 26 15:30:30 2020 -0400

Skip patching during SX data migration
    
    For SX upgrades we will skip applying the patches present in the
    upgrade_data. The N+1 patches are not applied after load-import so would
    not be present in the upgrade_data. With
    https://review.opendev.org/#/c/745355/ all N+1 patches applied to the
    system controller will be present after the subcloud install.
    
    Change-Id: I8f174211d8a4ddf4d84502d9c81736aaa22bc0c8
    Story: 2007403
    Task: 40759
    Depends-on: https://review.opendev.org/#/c/748288/
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>

commit 8a14160c5396a190a4dc75db7f8e65f81ddbd3c8
Author: John Kung <john.kung@windriver.com>
Date:   Tue Aug 25 09:05:58 2020 -0400

Update upgrades and patching get client to ensure token valid
    
    The client access for upgrades and patching are updated to
    get the client with cached keystone token on each invocation.
    
    This ensures that the cached keystone token expiry is checked
    each time, and if necessary, refreshed.
    
    Tests Performed:
    Distributed Cloud patch orchestration
    Distributed Cloud upgrades orchestration
    Verify by shortening and forcing token expiry on long running tasks
    
    Change-Id: I4f69c16e3a61bdbdfd717f135681b82e9d860425
    Story: 2007267
    Task: 40741
    Signed-off-by: John Kung <john.kung@windriver.com>

commit fa09faf22994b37ee90c30a1e68a9567c835e074
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon Aug 24 15:13:30 2020 -0500

Update DC subcloud API unit tests for proper validation
    
    The DC subcloud API tests were mocking away many of the validators,
    and were testing generic failure for most invalid inputs.
    
    Now the unit tests for subclouds API endpoint will check success
    and failure scenarios more accurately.
    
    Change-Id: Ic0fd4c1d9f3552601f95bf5031f94982985019c3
    Story: 2007267
    Task: 40800
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit b442f2afcdadb4e10893b66c085a9690005f5946
Author: John Kung <john.kung@windriver.com>
Date:   Wed Aug 19 12:06:28 2020 -0400

Enable subcloud install from image in dc vault
    
    The subcloud install is updated to allow reference to the
    vault load iso.
    
    When the image is not specified in the install values, or the
    image specified matches the vault load location, the image input iso
    is updated to reference the System Controller load vault.
    In such case, the bootimage iso is no longer duplicated, thus reduces
    the storage requirements.
    
    The -a' or '--active' flag, enables the upload of an active image:`
    system  --os-region-name SystemController  \
        load-import -a <bootimage.iso>  <bootimage.sig>
    
    Tests Performed:
    Import active image into dc vault
    Import active image into dc vault with new image
    Subcloud Install from image in dc vault
    
    Change-Id: I9faab0a5c9c7be7861ce2b4bc785cd44b9d4771a
    Story: 2007267
    Task: 40695
    Depends-On: https://review.opendev.org/#/c/746972
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 7f25a7141d08718c111aad54141b26d34555be19
Author: Bin Qian <bin.qian@windriver.com>
Date:   Fri Jul 24 21:22:23 2020 -0400

Trigger cert-mon audit from subcloud state changes
    
    When subcloud is managed, or becomes online, inform cert-mon via RPC
    call, so that cert-mon to perform endpoint certificate audit.
    
    Story: 2007347
    Task: 40442
    
    Depends-on: https://review.opendev.org/#/c/742468
    
    Change-Id: I733d2e22c2a4f7089b903b428025d87cee5123c4
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit ed2ffb4f6b280cdaf0cec1d5625ed5c88d5ebdfb
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Aug 12 08:43:01 2020 -0400

Delete N load as part of DC orchestration
    
    In this commit, a new step is added to remove the N load
    after the subcloud has been upgraded to N+1 release. The
    importing load step is also updated to remove an old load
    in the subcloud should one exist prior to importing the
    N+1 load.
    
    Story: 2007403
    Task: 40656
    Change-Id: If015b94354d47931c2ac9090b6c10c29ee051c3c
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit fa3e10c7e4667cfd70c9dbc069e1a55b7a15f8fb
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Aug 13 18:19:20 2020 -0400

Add subcloud system health check to pre_check
    
    Subcloud upgrade can only start if system health check passes,
    i.e. subcloud node(s) are all unlocked, enabled, patch and config
    current; all critical components are healthy and there are no
    management affecting alarms.
    
    The commit also includes the fix for a minor bug related to no data
    install value.
    
    Story: 2007403
    Task: 40665
    Change-Id: I6464a20ea82f5ad0d6ac5e2bdc93bbcb28e08b6c
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit a93baec6fe472de8da5fa01aae42153d4087c6b5
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Aug 19 21:07:24 2020 -0400

Add support for rd.net.timeout.ipv6dad to subcloud install
    
    The rd.net.timeout.ipv6dad boot option instructs dracut in the initrd
    to use the specified timeout for its IPv6 Duplicate Address Detection,
    rather than its default of 50 seconds. This allows the user to
    configure an increased timeout for a subcloud that may take longer for
    interface networking setup to be complete.
    
    Story: 2007267
    Task: 40712
    Change-Id: Ic7fe8eca975625f8634fff310de3d3550d7f5450
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit 3360a8d3cb142ceb35a3bb17f691baa8a60929d2
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Aug 13 13:59:17 2020 -0400

Fix system commands in dcorch-sysinv-api-proxy
    
    The request body for the system commands was not accessible
    when processing the response. The fix is to make a copy
    of the request and its body prior to processing the request.
    
    Closes-Bug: 1891546
    
    Change-Id: I2a6202311d0f9f6fbb91ae9581e8818c69c78587
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit cc43aa6205a4adfffb289556304602be64db52af
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu Aug 13 13:15:40 2020 -0400

Exception handling when subcloud not found
    
    When subcloud alarms are fetched for a subcloud name that doesn't
    exist, an exception is raised, but this exception is not generated
    correctly. This makes debugging the issue more difficult. A fix has
    been made to generate the error properly.
    
    Change-Id: I7f2efb94d3063980e1d7edc8aa6b9a19eea2f5e1
    Closes-Bug: 1884575
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 82fa2e9b2fa70712146de4ba563c91b74d31b2f4
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Aug 11 11:44:02 2020 -0500

Set subcloud firmware in-sync immediately after subcloud completes
    
    When a subcloud completed, it might show out-of-sync until
    the remaining subclouds were also orchestrated since that
    is when the audit is explicitly called.
    
    Now we update the status as each subcloud completes.
    
    Change-Id: I0694c7aacfb781aae1a7d22dd777fa75dff2bdfc
    Closes-Bug: 1890952
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 589ab369f398c8a8283a64d19b742fefa19a9f22
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Jul 31 11:20:20 2020 -0400

Fixes to support DC FPGA Orchestration
    
    When dcmanager-audit is restarted, it ensures that all endpoints
    exist for all subclouds. If a subcloud without the endpoint is
    encountered during startup, it adds an entry for it without
    requiring the subcloud to be re-added. A flag is also added
    on a replicated filesystem to avoid these checks from
    being executed more than once on the system controller.
    
    Change-Id: I049404c6a3e881623f3ff905a859a94f03bcf8e1
    Story: 2007875
    Task: 40471
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 663fcf0f8868c57540e682d47b93aab3d7a0c12c
Author: albailey <Al.Bailey@windriver.com>
Date:   Fri Jul 31 16:12:46 2020 -0500

Fixes for DC firmware orchestration related to labels and upload
    
    The applied image labels on the system controller should be the labels
    used when applying the image on the subcloud.
    
    The stage that removes images now uses all the applied labels
    for the image when removing them.
    
    The code that converts image fields for upload was passing an integer
    for the revoke_key_id.  It now converts to a string to prevent an
    encode exception.
    
    This commit also enables some unit tests for DC firmware
    orchestration that were being skipped.
    
    Closes-Bug: 1890355
    Closes-Bug: 1890340
    Closes-Bug: 1890324
    Change-Id: I1a730b01c0bbaeb5f3e1f6ecfb19f649911da453
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 5aab8597d40856f115e238f18a283d2964588e39
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu Aug 6 16:06:01 2020 -0400

Trigger firmware audit immediately once subcloud is managed
    
    When a subcloud is managed, patch audit and firmware audit
    are triggered. However, the patch audit on completion resets the
    force_firmware_audit variable, which prevents the firmware audit
    from immediately being triggered and keeps the subcloud out-of-sync
    until the next firmware audit cycle (Once in 1 hour). Thus, a fix
    is added to address this issue.
    
    Change-Id: Ie1aa2c7b4cb706d32b17f33bffa17d1004721887
    Closes-Bug: 1890294
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 749f7dc3d96f2842cd80b59d8022ae433e376494
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Fri Jul 24 10:52:23 2020 -0400

Add REST API timeout to patching and sysinv clients
    
    The REST API calls from the distributed cloud should timeout to
    prevent process to hang forever.
    
    Change-Id: Iea7f05018c348a817758f6bc920489127242ed11
    Closes-Bug: 1886037
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 537d2a9749337fdec501d834f1a84021e5bbd56a
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Fri Jul 31 09:23:38 2020 -0400

Extend default periodic interval for resource sync audit
    
    This commit extends the periodic time interval for resource sync audit to
    prevent the elapsed time of the dcorch audit exceeding the previous 10
    minute period along with the increment of a large number of subclouds.
    
    Change-Id: I96c93aef28c6b6359a03f652c6bb4618220a28db
    Closes-Bug: 1889506
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 8dcf8967da9f856e49b5038e5372d6b55d4326c6
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jul 30 17:57:14 2020 -0500

Fix issues in DC firmware orchestration
    
    Labels were not being properly passed in to the apply image API.
    
    If removing labels, a syntax error could cause the importing
    firmware state to fail.
    
    Change-Id: Ia5c378f373668c9b9e82b0b422af15deddd42849
    Story: 2007875
    Task: 40468
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 2b953fd0c7d8d4973d179df34d5497492a56c9f7
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Jul 30 19:07:31 2020 -0400

Add --force option when managing a subcloud
    
    When managing a subcloud, allow a --force option to be
    specified. The force option will allow an offline subcloud
    to be managed.
    
    This is necessary in cases where the user has (mistakenly)
    unmanaged a subcloud that is offline and the subcloud
    cannot be recovered while it is unmanaged. One example is
    a failure during DC upgrade orchestration that leaves the
    subcloud offline. To rerun upgrade orchestration and
    recover the subcloud, the subcloud must be managed.
    
    Story: 2007403
    Task: 40466
    Change-Id: I864de63b40b1065ffb1c6a025b5b221b793fbaf7
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 74027ac73986dc9eb8806ba3ebd836c6ba5a5194
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jul 8 15:44:13 2020 -0500

Add fw-update strategy type support for DC orch
    
    Firmware update states are:
     - 'importing firmware'
     - 'creating vim strategy'
     - 'applying vim strategy'
     - 'finishing fw update'
    
    Change-Id: I6f39f45a8b0f9d6b1ed8d33abb49715bb11a8496
    Story: 2007875
    Task: 40421
    Co-Authored-By: Jessica Castelino <jessica.castelino@windriver.com>
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 54ef00192d59b43c104d94229b4d66ea0a11b27e
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Thu Jul 23 08:06:07 2020 -0400

Add master reauthentication for identity data sync
    
    This fixes Issue #4 from Bug 1870999 where some subclouds failed to sync
    an identity resource change.  This was due to unauthorized request error
    on the master cloud due to expired token.  The current error handling
    only covers reauthentication of subcloud dbs client.
    
    In addition, the number of sync tries is increased from 2 to 3 to
    accommodate the case were data sync fails authentication on master and
    subcloud.
    
    Also fix exception in dcdbsync on subcloud due to uninitialized variable
    when handling change to project resource from dcorch audit on system
    controller.
    
    Change-Id: Ife8d86481a64ed841f5231a6df055028ee6597b0
    Partial-Bug: 1870999
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit 598178e36ed6039d2064a81cbd3adde3a51edddd
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Jul 16 14:10:25 2020 -0400

Support device image commands in dcorch-api-proxy
    
    The dcorch-api-proxy is updated to support the
    device image upload, apply, remove and delete operations.
    
    Story: 2007875
    Task: 40371
    Depends-On: https://review.opendev.org/#/c/742457
    
    Change-Id: Id5506226297e723f19b7d03ca18cc47ddfa8af40
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit 6fd2b44e750cec2949c7adff737b9c89dae7270f
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jul 21 17:21:21 2020 -0400

Firmware audit
    
    As part of the dcmanager, there will be a periodic audit which
    scans a number of subclouds in parallel and checks whether the
    subcloud has all of the “applied” device images that it should
    have (based on the labels the images were applied against and
    the host labels in the subcloud), and whether all of the “applied”
    device images have been installed on the hosts that they should
    be. If either of these is not true, then the subcloud
    “firmware image sync status” is considered “out of sync”. This
    will result in the subcloud as a whole being considered “out
    of sync”.
    When a device image is “applied” in sysinv, dcmanager
    will be notified and will set the “firmware image sync status”
    to “unknown” for all subclouds since it does not know at this
    point which subcloud(s) the image needs to be created or applied
    or updated in. On the next audit, this sync status will be
    updated to “in sync” or “out of sync” as applicable.
    
    Change-Id: Ifbbbeaba0bcd2691332e23d288856b8417c9efa8
    Story: 2007875
    Task: 40420
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 9a4ad4921a56556eb47968257f80da49cb2e586b
Author: Andy Ning <andy.ning@windriver.com>
Date:   Fri Jul 24 14:19:34 2020 -0400

Fix subcloud slow initial sync caused by keystone reauthentication
    
    Under certain conditions subcloud keystone client reauthentication
    will fail during initial sync. It will keep on failing in reattempts,
    causing endpoints to go in-sync slow after the subcloud is managed.
    
    This is caused by the keystone reauthentication reusing the existing
    token that is already invalid. Once that existing token expires
    (an hour at maximum), the reauthentication will succeed and endpoints
    go in-sync.
    
    This update fixed this by recreating subcloud admin session (and
    keystone, dcdbsync clients) by using username/password. With this
    the reauthentication will cover cases where subcloud admin session
    user changes (eg, from admin to dcmanager or vise versa).
    
    Change-Id: I8a49bf8c55e3538fc47b833ae648b667b5e9e9e5
    Closes-Bug: 1887849
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit 0620b421fd7999ab37747114c0a0f5ec8a098dd5
Author: John Kung <john.kung@windriver.com>
Date:   Tue Jul 28 09:28:22 2020 -0400

Update dcmanager upgrade orchestration client access
    
    During upgrade orchestration with long running operations, it is
    possible that the keystone token may expire and result in
    unauthorized http errors.
    
    In dcmanager upgrades orchestration, the client usage is updated to
    retrieve the (cached) keystone client to ensure the checks for the
    token expiry are up to date.
    
    The unauthorized http exception handling is made explicit for the long
    running client operations.
    
    Change-Id: I281ecc51b1a4349ac865f0c4f6ea8244b7cb1fcc
    Story: 2007403
    Task: 40452
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 49f7ad12fa61430842d7da8663ec9dc8c7378f57
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Jul 22 12:25:10 2020 -0400

Update DC ICA expiry duration to 1 year
    
    The current 10 years expiry duration is incorrect. Change it to 1 year.
    
    Change-Id: Ia9cb8c104c1bb2b1028b29ea1cb39c8aea9d235a
    Closes-Bug: 1888521
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit 73998010b58073a15bc99bf58928376433e10937
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Jul 13 21:49:57 2020 -0400

Add recovery code for failures during each upgrade state
    
    In this commit:
      - Pre check for online case is updated. If the subcloud
        deploy status is 'migrated', advance directly to activating
        upgrade step. Otherwise, start from the first step
        (installing license).
      - The subcloud deploy status is set to 'migrated' at the end
        of migrating data step.
      - The subcloud deploy status is set to 'complete' at the end
        of completing upgrade step.
      - Skip data migration if deploy status is either 'migrated'
        or 'complete'.
      - Reduce the number of retries for failed activation from 45
        to 10. The overall wait time is still 45 minutes.
      - Improve error messages to user.
      - Add another log to give clarity to the time it takes the
        subcloud to respond to load import request.
      - Various small fixes.
    
    Tests:
      - Load import failure - not enough disk space
      - Load import failure - bad load file
      - Upgrade start failure - sysadmin directory too large
      - Controller lock failure - processing failure
      - Upgrade simplex failure - incorrect bmc password
      - Upgrade simplex failure - network glitch
      - Data migration failure - network glitch
      - Upgrade activation failure - processing failure
      - Upgrade completing failure - processing failure
    
    Story: 2007403
    Task: 40351
    Change-Id: Idf0bf8009122018e4597a7f1dede58658d0a064b
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit f0dd6b9101ec28ef76faa0bf4b422345cf1d7bbc
Author: John Kung <john.kung@windriver.com>
Date:   Mon Jul 20 17:29:35 2020 -0400

Add prechecks for data_install and scratch filesystem
    
    As part of precheck state checks, check data_install values and
    subcloud host scratch filesystem size.
    
    The data_install values are semantically checked on subcloud update,
    so the precheck verifies its presence.
    
    Tests Performed:
    Verified data_install empty check
    Verified subcloud scratch host filesystem check when online
    Verified pre_check stage continues to next stage on success
    tox unit tests updated and passed
    
    Change-Id: I972309f08d648cad6719b9a340a876258f6f912a
    Story: 2007403
    Task: 40396
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 43492c97db804e6a88053b1abd63d4ddf35749e4
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Fri Jul 17 16:55:51 2020 -0400

Add --force option to strategy create API
    
    The new option will allow the user to retry the upgrade of the
    specified subcloud that previously failed during remote install
    or data migration step.
    
    Story: 2007403
    Task: 40387
    Change-Id: Ib17857340c85892ff962bfcf3f9793c4e65bbef2
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit dc6d66c969c9236890013f2a029c8e65d422bac5
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Jul 8 17:06:16 2020 -0400

Add RestAPI to update subcloud sync-status
    
    Currently set restriction to only allow updating endpoint dc-cert
    Updating other endpoint status can be enabled when there is use case.
    
    Passed TCs:
    Test with testing client to
    1. successfully update dc-cert to in-sync, out-of-sync and unknown
    2. updating endpoint other than dc-cert is rejected
    3. updating dc-cert status to word other than above mentioned is
       rejected.
    
    Change-Id: I77df581c9d1d461f72948499e9410f22819b1030
    Story: 2007347
    Task: 40300
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit 6a16d05061e23c87c53b2ca6e89a2b383db8f850
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Wed Jul 15 23:53:10 2020 -0500

Fix dc-cert sync status handling for unmanaged subcloud
    
    When a subcloud is unmanaged and then managed again, it will
    never go back in sync. This is because all sync_statuses are
    set to unknown when the unmanage is done. However, the
    dc-cert_sync_status is only set to in-sync when the subcloud
    transitions from offline to online. So in this case the
    dc-cert_sync_status stays as unknown.
    
    The fix is to avoid setting the dc-cert sync status to unknown
    when a subcloud is unmanaged. The dc-cert sync status will
    still being audited for unmanaged subclouds so there is no
    need to do this.
    
    Change-Id: I7a53c11888b8866c6077679265d69c36faf4e264
    Closes-Bug: 1887733
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 6408de3d7e36c8837cb7b37c670455cb80efca61
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue Jul 14 09:48:20 2020 -0400

Reduce the patch audit interval
    
    Currently the patch audit assumes it is waking up
    every 20s so it will wait through 300/20 audit cycles.
    Should the subcloud audit take more than 30 seconds,
    the patch audit only runs about every 13-14 minutes.
    
    This update stores a timestamp for the last patch audit
    and compare with the current time when deciding if an
    audit is required. In addition, it triggers the load
    audit when a patch audit is forced.
    
    Story: 2007267
    Task: 40342
    
    Change-Id: I29e096fd3f2c82f42263f737390b64452bb606c1
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 91e79f1d608489c39025695689343458e18da5c9
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jun 24 12:23:51 2020 -0500

Add a 'type' filter for update strategy API calls and DB filtering
    
    The CLI and manager classes can pass an additional 'type' filter so
    that upgrade and patch strategy commands are independent of each other.
    
    This allows the patch orchestration CLI to only show and have control
    over patch strategies. Horizon will also only show and manage patch
    strategies.
    
    Note: there can still only be one strategy in the DB at one time.
    When the api filter attribute for the update type is not passed to the
    URL, any strategy type can be queried.
    
    The 'create' and 'show' commands explicitly indicate when a mismatched
    strategy type is present.
    
    Change-Id: I40058b8cd5d09fa2b700f0a45d10a6b59b74778c
    Story: 2007403
    Task: 40348
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 4e4355458d0f680bf150032000e1ca9b84682e2d
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jun 17 11:05:01 2020 -0500

DC upgrade state machine states will select next state
    
    Rather than having the state machine states as an ordered list,
    this modification allows each state to specify the next state.
    
    This enhancement allows future features like compound states
    or a pre-check state to enable skipping over known states
    or alternative transitions.
    
    This adds a pre-check state. Only the success path has been tested
    so far.
    
    Change-Id: Ideae9885b6026cb441bcb6ac08204784b876f3a9
    Story: 2007403
    Task: 40347
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit b5e45f15fe697295b18c62c678c745b3d798cd77
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Jul 13 21:58:50 2020 -0400

Update resources purge logic
    
    The daily dcorch database purge cron job deletes resources that are
    no longer used by any orch jobs but as the subcloud_resource table is
    set to cascade delete it also removes subcloud resources that are
    still in use. This is then flagged by the next dcorch audit and the
    subcloud resources are re-created over the next audit cycle(s).
    
    In this commit, only purge resources that are not referenced by any
    orch jobs and subclouds.
    
    Closes-Bug: 1887430
    Change-Id: I7179a2fba21f296dd51660bb224f5d51aaf531d8
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit 3941475e5f17998b6ddbc3426be46db65135ac98
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed Jun 24 13:50:11 2020 -0400

Subcloud-group based orchestration
    
    Whenever an orchestration strategy is created, it will ensure that
    application of patches or software upgrades would maintain the below
    ordering for the subclouds :
    1) All subclouds in default group first.
    2) Then all subclouds in the “first” group, then the “second”, etc…
    3) Never include subclouds from different groups in the same “step”
       of the strategy, to ensure they are not patched at the same time.
    4) If max_parallel_subclouds and subcloud_apply_type are provided while
       creating strategy, it will supercede the max_parallel_subclouds and
       update_apply_type values set for each subcloud group
    5) If max_parallel_subclouds and subcloud_apply_type are not provided,
       we must respect the max_parallel_subclouds and the serial vs.
       parallel choice set for each subcloud group.
    
    Change-Id: Ifc98e303f70716f3c7e528167658617a6455ca65
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Story: 2007403
    Task: 40164

commit 241ad4b31df13d0026c2c987e75a5e84b5d1d86d
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jul 7 12:20:38 2020 -0500

Allow audit for additional deploy status failed states
    
    Adding the deploy status failure states to the list of auditable
    criteria for subclouds.  This allows subclouds to be set to
    offline by the audit rather than skipping the audit for them.
    
    Change-Id: Ia4ce5b36c7e108afd8da98412ee4bc8c29c3a49c
    Closes-Bug: #1885817
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 4f5d0b4fca0d9494f3acfd32349848558a9c34eb
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Tue Jun 30 11:50:15 2020 -0400

Return master_id from get_resource_id for deleting SNMP resource
    
    This commit modified the get_resource_id method in SysinvSyncThread.
    The previous code only return community/ip_address which is for the
    creat service operation. By this change, the get_resource_id method
    in this class will return resource.master_id for deleting SNMP
    community and trapdest.
    
    Test:
      1 Add snmp trapdest and snmp community configuration from system
    controller, with command:
        system snmp-comm-add -c test1
        system snmp-trapdest-add -c test1 -i 10.11.2.1
      2 Wait the snmp community and trapdest configured in subcloud in
    subcloud controller:
        system snmp-comm-list
        system snmp-trapdest-list
      3 Delete the snmp community and trapdest in system controller:
        system snmp-trapdest-delete 10.11.2.1
        system snmp-comm-delete test1
      Expected: trapdest or community should be deleted from subcloud,
    result can also be observed  in dcorch DB and /var/log/dcorch.log.
    
    Change-Id: I8d3b40599f83263cade1799d5a1dae72d3bac2c5
    Closes-Bug: 1863045
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 4f1c5db8094fc82b8b1defb5e574cb3e54cd4035
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 30 14:08:43 2020 -0400

Sync only ‘ssl_ca’ and ‘openstack_ca’  certificates to subclouds
    
    The 'docker_registry', 'tpm_mode', and 'openstack' cert on the
    SystemController will have a CN and SAN with OAM IP Address, any
    related DNS Names for OAM IP Address and MGMT IP Address. If these
    certificate are synched to the subclouds, even if they are signed by
    a trusted CA, when any client connects to the subcloud OAM or MGMT
    IP, the certificate that comes back will not pass certificate
    validation because the certificate does not apply to the subcloud's
    IP Address. For this reason a fix is put in place to avoid only sync
    "ssl_ca" and "openstack_ca" certificates to subclouds.
    
    Change-Id: I17ac94d79beb04f559c46062dbe7826590fcdb06
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1865643

commit 0d1799d810943d9d651ef919ca23f23f56c91d9d
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jun 30 13:44:30 2020 -0500

Increase the DC orch upgrade-activation timeout
    
    Testing on one subcloud took 19 minutes for the upgrade
    activation to complete, and so it timed out.
    
    The timeout has been increased from 15 minutes to 45 minutes.
    
    Change-Id: Ice69acbbe097ea038e2932edb44f59f024cbeefb
    Closes-Bug: 1885451
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit cfd225b75bdf355b5df7c0125078dd15c03584b5
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jun 25 09:28:31 2020 -0500

Adding a wait to the end of migrate_data state
    
    The ansible script triggers a reboot and the state needs to
    wait for the reboot to complete, before the state transitions
    to the next state.
    
    This adds a retry for the upgrade-activate step, and increases
    the total max time for that task to be 15 minutes
    
    This also fixes a logging bug where a failure in one subcloud
    causes excessive logs while other subclouds are being processed.
    
    Closes-Bug: 1885132
    Change-Id: Iabca2ecb0bacff7c5622c90722fcfb938a71a768
    Story: 2007403
    Task: 40172
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 8ecba753119da83e80f4933bd24dac311b5979f1
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Thu Jun 25 14:16:45 2020 -0400

Add timeouts to DC sysinv and patching clients
    
    If a subcloud is powered off or loses communication with the system
    controller while the subcloud audit is waiting for a response to a
    sysinv or patching request, the subcloud audit will hang forever.
    Add some failsafe 10 minute timeouts to ensure the audit can recover.
    
    Change-Id: I369225119f89ab5a2f05137aba955dda982d51c5
    Closes-Bug: 1884560
    Closes-Bug: 1870413
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit dd9c780663af60666e82d2441e242dd6ca1e72f9
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue Jun 23 17:02:19 2020 -0400

Add CLI/API to populate data_install
    
    The dcmanager subcloud update API is extended to
    allow populating/updating the data_install values required
    for remote install (upgrade to stx 4.0):
    
    In addition, the bmc-password is stored as base64encoded.
    
    Test cases:
    1. Add a subcloud without install-values and update the
       subcloud with install-values
    2. Update the subcloud install-values with new data
    3. Verify the bmc_password is stored as base64encoded
    4. Add a subcloud with install-values and verify the
       bmc_password is populated in rvmc.conf as base64encoded
    5. Subcloud update with the previously supported parameters
    
    Story: 2007403
    Task: 40149
    
    Change-Id: I78db37873547fde43f085a0093765f838d2e4200
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit a331e544fa16b8cee20c7cc392da1b34805e6c1e
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 23 11:31:16 2020 -0400

Distributed Cloud re-execution susceptible to old data
    
    Added an os_O_TRUNC flag to ensure that files are truncated
    before writing new content to it. This avoids previous content from
    being appended to the new file.
    
    Change-Id: I14438d4276376089c1c46a20626f251adcac09f9
    Closes-Bug: 1883113
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit e422388185102616b35946a6bd04d038e415380a
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jun 11 15:22:22 2020 -0500

Add thread loop breakout handling for DC upgrade states
    
    If a state machine handler has internal loops, it can check to
    see if the main thread is being stopped, and can break out early.
    
    The states performing this check are:
     - importing load (can take up to 30 minutes)
     - starting (can take up to 30 minutes)
     - lock controller (can take up to 10 minutes)
     - unlock controller (can take up to an hour)
     - activating (can take up to 10 minutes)
     - completing (can take up to 10 minutes)
    
    Added a loop-check for 'starting' state to wait for
    upgrade to move to 'started'.
    
    Added a loop-check for 'completing' state to wait for
    upgrade to be deleted.
    
    Change-Id: I2d4e77d1baf917738ddc96f3cbaa76249ba17fdf
    Story: 2007403
    Task: 40105
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 36dce2d5f072ba464e9cb1fe82e80bf14c6dc943
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jun 10 13:26:55 2020 -0500

DC upgrade state handler improvements
    
    1) DC upgrade importing-load state should wait for 'imported'
    
    The importing_load state for software upgrade orchestration was
    moving to the next state after the load was 'importing' instead
    of 'imported'.  This caused the next state to fail since it required
    the load to be in the imported state.
    
    2) Use the strategy options object 'alarm-restriction-type' to
    determine if an 'upgrade start' should use a force to ignore alarms
    that are not management affecting.
    
    3) Add polling stage after upgrade activation to wait for the activity
    to complete before transitioning to the next state.
    
    4) Update the unit tests to mock timeout values so that
    running the unit tests is quicker.
    
    Change-Id: Ibaf77d2c33b633f915301344c7a7318a59fe3a9f
    Story: 2007403
    Task: 40040
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 1012dd28964b7416e24b3dd60b217988d137e640
Author: John Kung <john.kung@windriver.com>
Date:   Tue Jun 9 10:22:56 2020 -0400

Implement the state for Simplex Upgrade
    
    The Simplex Upgrade step upgrades the simplex subcloud to the load
    imported in dc-vault corresponding to the target software release.
    
    Required static data is captured at subcloud add and referenced for
    the simplex upgrade.
    
    Dynamic data is retrieved from the online subcloud; including the
    bmc credentials which are obtained via a synchronized service within
    project allowing for access to the bmc credentials.
    
    BMC password retrieval endpoint_cache integration
    with https://review.opendev.org/#/c/736004/
    
    Tests Performed:
    - Subcloud upgrade (install) with load corresponding
      to System Controller
    - Subcloud add persists install data
    - Simplex upgrade persists dynamic upgrade data from start of
      upgrade simplex
    - Obtain bmc password from online subcloud
    - Obtain bmc password from install data, in case the
      bmc is not configured
    
    Pending for subsequent commit:
    - tox unit tests
    
    Change-Id: Ie047139280a5780bfe47b9f9959f4c6781d6f3fa
    Story: 2007403
    Task: 40024
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 94946613cdb51e1a7b9c08ad8344d120da0ed59f
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Fri Jun 12 13:14:57 2020 -0400

Implement simplex subcloud data migration state
    
    This commit implements the data migration state for subcloud upgrade
    orchestration.
    
    Story: 2007403
    Task: 40071
    Depends-On: https://review.opendev.org/#/c/734620/
    Depends-On: https://review.opendev.org/#/c/733902/
    Depends-On: https://review.opendev.org/#/c/733914/
    
    Change-Id: I5b38f31324ddb62b6760f897a1eb3cca5159e2ef
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit d10fef48217d69a45084dd8b81c30d3bbe68b3af
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue Jun 16 15:08:20 2020 -0400

Configure dcmanager user for endpoint_cache
    
    The following changes are made, in order to remove the
    dependencies to 'admin' user and use 'dcmanager' user to
    authenticate with services in the subclouds:
    . Add endpoint_cache section to both dcorch and dcmanager
    . Configure dcmanager user in the endpoint_cache section
    . Sync the dcmanager user to subclouds
    . Use 'dcmanager' user for subcloud authentication by default,
      and it falls back to 'admin' user if the authentication fails
     (This is required during software upgrade).
    
    Depends-On: https://review.opendev.org/#/c/735994/
    Partial-Bug: 1883758
    
    Change-Id: Ibc78abc86a7a825f83f2cac9fd54e4183e7ccd80
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit e8d916756d439cf63eed52f347a2a547fc5124cb
Author: Andy Ning <andy.ning@windriver.com>
Date:   Wed Jun 17 10:40:23 2020 -0400

Set up /var/run/dcmanager as dcmanager's default temp files location
    
    dcmanager call the k8s python client to perform a number of
    operations. The k8s python client creates temp files under /tmp and
    continues use these tmp files for the life-cycle of the processes.
    
    However systemd-tmpfiles-clean.service will run every day to clean up
    files in /tmp dir that are older than 10 days. If the k8s client code
    is not triggered for more than 10 days (thus its temp files are not
    accessed for more than 10 days), these temp files will be removed as
    part of the cleanup. Certain dcmanager operations then starts to fail
    with an error that the tmp file is no longer there.
    
    This is a known issue of kubernetes python client:
    https://github.com/kubernetes-client/python/issues/765
    
    The commit fixes this issue by setting TMPDIR to /var/run/dcmanager
    when sm starts dcmanager-manager.
    
    Change-Id: Ib147c2ab26e303032e18da51a506e3768bc471e0
    Closes-Bug: 1883599
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit dee5e270a3752ac47f4282cb12a5c273e093068b
Author: Bin Qian <bin.qian@windriver.com>
Date:   Tue Jun 16 16:51:55 2020 -0400

Ensure subcloud cert data is intact
    
    In reported cases subcloud bootstrap only received tls.key. With tls.crt
    and root_ca_crt are both empty.
    Solution is wait for the data values are not empty.
    
    Change-Id: I01d71f5d000247f03d2a32eb37761f3e186ac863
    Closes-Bug: 1883777
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit ed652edb38414343528bd78378fde06f7af5b66a
Author: Bin Qian <bin.qian@windriver.com>
Date:   Fri Jun 12 10:50:09 2020 -0400

Update dc-cert subcloud status endpoint when subcloud online
    
    Set it to in-sync and will be replaced with dc-cert validation
    result (coming soon).
    Ensure updating dc-cert status before subcloud is unmanaged.
    This is because the subcloud is bootstrap with in-sync dc-cert.
    And dc-cert is audit when subcloud is online (before set to
    managed).
    
    Change-Id: I2e6aa597e53da5e26f44430767eb6fd4d3e784ee
    Story: 2007347
    Task: 40070
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit c1da0762d58182171c12aca783c8f00c9ef2ac2b
Author: Sharath Kumar K <sharath.kumar@intel.com>
Date:   Tue May 5 06:28:22 2020 +0200

Tox and Zuul job for the bandit code scan in stx/distributedcloud
    
    Setting up the bandit tool for the scanning of HIGH severity issues
    in the python codes under Starlingx/distributedcloud folder.
    Expecting this merge will enable zuul job for CI/CD of bandit scan.
    
    Configuration files:
    1. tox.ini for adding bandit environment and command.
    2. test-requirements.txt for adding bandit version.
    3. .zuul.yaml file for adding bandit job and configuring under
       check job to run code scan every time before code commit.
    
    Test:
    Run tox -e bandit command inside the fault folder to validate the
    bandit scan and result.
    
    Please note:
    Changes will be implemented in batches and  this is Batch6 change.
    
    Story: 2007541
    Task: 39648
    Depends-On: https://review.opendev.org/#/c/721294/
    
    Change-Id: I6fdfeacdb000faafc73c44eef352bf2abd38a9a2
    Signed-off-by: Sharath Kumar K <sharath.kumar@intel.com>

commit cd12fc31a1702f08f77fedcaa6386825d9ae3c6c
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jun 11 08:43:21 2020 -0500

Add reboot handling to unlock state for DC upgrades
    
    When an unlock is triggered, the host is rebooted.
    
    The maximum amount of time for this state to complete is 1 hour.
     - Reboot time (30 minutes)
     - pre and post reboot time (30 minutes)
    
    The unlock state handler for upgrade orchestration will
    now loop and ignore failures for a period of time, while
    waiting for the host to become unlocked / enabled.
    
    Change-Id: Ie7ad8dbfecd6ffbe241d098c8677a8e25f400b07
    Story: 2007403
    Task: 40048
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 4fd65e9913d28bd097b16ad3bfb80c04ca6b5419
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon May 25 13:51:18 2020 -0400

CLI command to deploy a subcloud
    
    If deployment failed, the user has no option than to delete and
    re-add it. If the user was re-adding the subcloud without
    re-installing, it would further result in a bootstrap failure.
    Thus, to simplify things, a new CLI command is provided to allow
    re-deployment. Furthermore, if the user still chooses to delete
    the subcloud and re-add it without a re-install, a better error
    message is provided asking them to re-install the host.
    
    CLI:
    dcmanager subcloud reconfig <id/name> --deploy-config <file>
    
    Test Cases:
    1) Successfully add a subcloud with or without deployment option
    2) Fail to re-add a subcloud without re-installation after a failed
       deployment
    3) Re-deploy with new CLI command after successful and unsuccessful deployment
    4) Re-deploy with new CLI command before and after the subcloud is unlocked
    5) Test new CLI command by passing wrong parameters
    
    Change-Id: I9fe7e3791e3887160668281048c3c12a7f40c2af
    Partial-Bug: 1864756
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 66f9341692336e86ee74712dc05e816fae5397ba
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 9 00:34:14 2020 -0400

Manage subcloud only when deploy_status is "complete"
    
    Subcloud can be managed only if the deploy status is "complete"
    i.e. either subcloud is successfully bootstrapped or subcloud is
    successfully deployed. This will prevent partially configured
    subclouds from being managed and put into service.
    
    Change-Id: Ic4562c250e651f6df64f368db641d7c58aee887c
    Closes-Bug: 1882786
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 5e5c416a9b56d6c7385e6883a73c8b29ebce4709
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jun 2 15:51:32 2020 -0500

Implement import-load and starting-upgrade strategy states
    
    This commit allows the state machine to be defined through an
    ordered list of states. The states are mapped to by a dictionary
    to state handler classes.
    
    This greatly simplifies the software upgrade orch thread so that it
    can be refactored in the future to share functionality with other orch
    thread strategy types.
    
    The following states have been implemented:
     - 'importing load'
     - 'starting upgrade'
     - 'activating upgrade'
     - 'completing upgrade'
    
    The 'installing license' state now uses this design pattern.
    Its functionality and unit tests are now in their own files.
    
    The 'unlocking' state requires additional changes for the
    'activating' and 'completing' states to pass.
    
    Change-Id: Iedc40bdea406461dd4c2ce3887a37be6ec3c999f
    Story: 2007403
    Task: 40001
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 70f4dc8537ae89cf2eb70a78bda64073f13b02a1
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 9 13:43:04 2020 -0400

Subcloud deploy fails
    
    Deploy config generated by dcmanager does not truncate the existing
    file after it has been overridden. This leaves some invalid content
    at the end of the newly generated file. Fixes have been made for this.
    
    Change-Id: I6c7e7fbacb1ee323c3675076083e046498698425
    Closes-Bug: 1882826
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit efcfc7a45a170c51ea5094fd62cb7047b24fc92d
Author: Andreas Jaeger <aj@suse.com>
Date:   Thu Jun 4 14:16:23 2020 +0200

Switch to newer openstackdocstheme and reno versions
    
    Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
    these versions will allow especially:
    * Linking from HTML to PDF document
    * Allow parallel building of documents
    * Fix some rendering problems
    
    Update Sphinx version as well.
    
    Disable openstackdocs_auto_name to use 'project' variable as name.
    
    Change pygments_style to 'native' since old theme version always used
    'native' and the theme now respects the setting and using 'sphinx' can
    lead to some strange rendering.
    
    openstackdocstheme renames some variables, so follow the renames
    before the next release removes them. A couple of variables are also
    not needed anymore, remove them.
    
    See also
    http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
    
    Change-Id: I7da2bdaafe86b9eb930202bef3742571daa90e17

commit 125e465aff6801940f25762d7be2406d01349d18
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon May 25 12:16:55 2020 -0400

Extend sysinv api proxy to support load operations
    
    In this commit, the dcorch-sysinv-api-proxy is extended
    to support load import and delete requests.
    
    Upon receiving a successful response from sysinv api for load
    import/delete request, the load is saved to/removed from dc-vault
    accordingly.
    
    Tests:
      - Successful load import
      - Successful load delete (no subclouds have the deleted load)
      - Successful load delete (one subcloud has the deleted load)
      - Failed load import (exceeding limit)
      - Failed load import (bad signature)
      - Failed load delete (does not exist)
    
    Unit tests will be added via a separate commit under task 39903
    of story 2007082.
    
    Story: 2007403
    Task: 39840
    Depends-On: https://review.opendev.org/#/c/730632
    Change-Id: If2769b0faf093523e7e9bc97b8cdc6a5513534aa
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 63c007461d782158e62aa082d74c44c452969883
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon May 25 12:55:20 2020 -0500

Implementing Lock and Unlock upgrade strategy states
    
    Implement the states for lock and unlock.
    
    Adding stub methods and classes for the full upgrade strategy
    subcloud orchestration state machine.
    
    Adding unit tests to validate all lock and unlock success
    and failure scenarios.
    
    Change-Id: I5b8d89bc733b867e8568043b0c7e4a79d1925890
    Story: 2007403
    Task: 39874
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 2f49219858ed3c3a872dcaa337ba601a68fce7e4
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue May 26 16:20:45 2020 -0400

Combine the patch audit with the subcloud audit
    
    This commit moves the patch audit to dcmanager-audit process.
    Patch audit does not run at the same frequency as subcloud
    audit, its normal interval is approx. 5 mins, but it can be
    forced to run on the next audit interval by calling
    trigger_patch_audit RPC.
    
    Change-Id: I4d4fdd0cebe9f3f65241107c92bbc4d26c13c8f3
    Story: 2007267
    Task: 39864
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 30f2147c398eb56fb67cb7b777560b3a18529ba5
Author: Tao Liu <tao.liu@windriver.com>
Date:   Mon May 25 13:40:16 2020 -0400

Close any temp files after upload a patch
    
    Add a cleanup routine in the patch-api-proxy to close
    any temporary files that opened by third-party libraries,
    during a patch upload.
    
    Verified that the temp files are close and /scratch
    usage is not increased after a few of large patch
    upload.
    Verified that the temp files are close and /scratch
    usage is not increased after a failed upload.
    
    Change-Id: I68453653fe09d8b1ad80289ab3ff33e476b3acee
    Closes-Bug: 1880587
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 566076a0a8808ee958f5edb4fb9333f426bf1490
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Wed May 20 02:30:03 2020 -0400

Fix slow subcloud manage due to sysinv auth
    
    Synchronize keystone services project and sysinv user ids between the
    system controller and subcloud during subcloud bootstrap.  This fixes
    issue where the subcloud would go offline on initial manage for 1 hour
    until keystone token expires.
    
    Change-Id: I2b36861df0858920eed308bd8e9a12b49b68a191
    Closes-Bug: 1877419
    Closes-Bug: 1877584
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit 6c09914c1140bd7d7e3988e7c42500c6d98df100
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 14 09:12:31 2020 -0500

Adding a license install step for subcloud upgrade
    
    The license install step queries the license on the
    system controller and installs it on the subclouds.
    
    If the system controller does not have a license,
    the license install state is skipped and the strategy
    proceeds to the next stage.
    
    On successful completion of the 'installing license' stage
    the strategy proceeds to the next stage which is the
    'importing load' stage.
    
    Change-Id: I61bc144bdf89302b4ba11fd35e82a9ff64d9c517
    Story: 2007403
    Task: 39780
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit bd4a76782d6b4896979c7c2fad467edc129b6ad7
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon May 18 17:40:18 2020 -0400

Use system software version for load audit.
    
    The current audit logic is based on the software version
    of the active load as opposed to the software version of
    the system. As a result, the subclouds will be out-of-sync
    shortly after the first controller has been upgraded.
    
    In this commit, the audit task checks the system software
    version which is only updated after both controllers have
    been upgraded. This will ensure no premature subcloud
    upgrade during upgrade orchestration.
    
    Tests:
      - Modify the system software version of the central
        cloud to simulate a software upgrade and verify that
        all subclouds are out-of-sync after the load audit.
      - Revert the above change and verify that all subclouds
        are back in-sync after the load audit.
      - Modify the system software version of one of the
        subclouds and verify that it is out-of-sync after
        the load audit.
    
    Closes-Bug: 1879396
    Change-Id: Ia10d561730f66f63f3e3113d8c3c56270051e7f3
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

auto-github-kubernetes-client-python #765
[closed lifecycle/rotten] Edit

Bug watches keep track of this bug in other bug trackers.