StarlingX

Distributed Cloud: Processes can hang due to missing REST API timeouts

Bug #1886037 reported by Bart Wensley on 2020-07-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Yuxing

Bug Description

Brief Description
-----------------
There have been two bugs where the dcmanager subcloud audit process hung (bug 1884560 and bug 1870413). The root cause was determined to be that the REST API calls made by the audit did not use timeouts, so a lost message could result in the process hanging forever. Timeouts for the REST API calls used by the audit were added as a fix for bug 1870413.

This LP is being opened to fix the remainder of the REST API calls in the distributed cloud code. Some notes:
- The DC orchestration code is also vulnerable to this issue as it uses the sysinv/patching/vim clients with no request timeouts.
- We need to protect all rest API requests we send with a timeout.
- This will also require changes to the nfv_client itself (in the VIM) because it currently doesn’t provide any way for users to specify a timeout.

Severity
--------
Major: processes can hang with the only recourse being to restart them

Steps to Reproduce
------------------
This is a race condition - it can be difficult to reproduce. It is usually triggered when a subcloud is powered down.

Expected Behavior
------------------
All REST API calls from the distributed cloud processes should time out - even in the case of message loss.

Actual Behavior
----------------
See above

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Distributed Cloud

Branch/Pull Time/Commit
-----------------------
See bug 1870413

Last Pass
---------
N/A

Timestamp/Logs
--------------
See bug 1870413

Test Activity
-------------
See bug 1870413

Workaround
----------
A controller swact will restart the affected processes

See original description

Tags:

Bart Wensley (bartwensley) on 2020-07-02

tags:

added: stx.distcloud

Bart Wensley (bartwensley) on 2020-07-13

description:	updated
Changed in starlingx:
assignee:	nobody → John Kung (john-kung)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-13: Fix merged to nfv (master)

Reviewed: https://review.opendev.org/740779
Committed: https://git.openstack.org/cgit/starlingx/nfv/commit/?id=b39bc8c0349e68373d30360c1d99ad02c7b262aa
Submitter: Zuul
Branch: master

commit b39bc8c0349e68373d30360c1d99ad02c7b262aa
Author: Bart Wensley <email address hidden>
Date: Mon Jul 13 14:01:27 2020 -0500

Add REST API timeouts to nfv-client

    The VIM's nfv-client sends REST API messages to the VIM, but it
    currently does not set a timeout for these calls. In the case
    where a message is lost, the nfv-client would block forever,
    causing the calling process to hang.

    Fixing this by adding timeouts to the nfv-client to ensure that
    all REST API messages it sends will timeout even in the case of
    lost messages.

    Change-Id: Iba06924fb7bd14a1ee3362b4fd19aa4114dc34cd
    Partial-Bug: 1886037
    Signed-off-by: Bart Wensley <email address hidden>

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-07-15:

stx.5.0 / medium priority - issue related to a race condition which is hard to reproduce

Changed in starlingx:
importance:	Undecided → Medium
tags:	added: stx.5.0

Bart Wensley (bartwensley) on 2020-07-21

Changed in starlingx:
assignee:	John Kung (john-kung) → Yuxing (yuxing)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-24: Fix proposed to distcloud (master)

Fix proposed to branch: master
Review: https://review.opendev.org/742929

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-05: Fix merged to distcloud (master)

Reviewed: https://review.opendev.org/742929
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=749f7dc3d96f2842cd80b59d8022ae433e376494
Submitter: Zuul
Branch: master

commit 749f7dc3d96f2842cd80b59d8022ae433e376494
Author: Yuxing Jiang <email address hidden>
Date: Fri Jul 24 10:52:23 2020 -0400

Add REST API timeout to patching and sysinv clients

The REST API calls from the distributed cloud should timeout to
prevent process to hang forever.

    Change-Id: Iea7f05018c348a817758f6bc920489127242ed11
    Closes-Bug: 1886037
    Signed-off-by: Yuxing Jiang <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-19: Fix proposed to distcloud (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/792298

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-27: Change abandoned on distcloud (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/792298
Reason: Updated merge soon

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-27: Fix proposed to distcloud (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/793405

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-15: Change abandoned on distcloud (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/793405

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-15: Fix proposed to distcloud (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/distcloud/+/796528

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-16: Fix merged to distcloud (f/centos8)

#10

Download full text (105.0 KiB)

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/796528
Committed: https://opendev.org/starlingx/distcloud/commit/4c5344f8765b372cb84d2b1181589c16db2ae6e4
Submitter: "Zuul (22348)"
Branch: f/centos8

commit cb979811017bd193fc1f06e53bb7830fd3184859
Author: Yuxing Jiang <email address hidden>
Date: Wed Jun 9 11:11:27 2021 -0400

Format the IP addresses in payload before adding a subcloud

    The IPv6 addresses can be represented in multiple formats. As IP
    addresses are stored as text in database, ansible inventory and
    overrides, this commit converts the IP addresses in payload to
    standard text format of IPv6 address during adding a new subcloud.

    Tested with installing and bootstrapping a new subcloud(RVMC
    configured) with the correct IPv6 address values, but with
    unrecommended upper case letters and '0'. The addresses are
    converted to standard format in database, ansible inventory and
    overrides files.

    Partial-Bug: 1931459
    Signed-off-by: Yuxing Jiang <email address hidden>
    Change-Id: I6c26e749941f1ea2597f91886ad8f7da64521f0d

commit 2cf5d6d5cef0808c354f7575336aec34253993b3
Author: albailey <email address hidden>
Date: Thu May 20 14:19:24 2021 -0500

Delete existing vim strategy from subcloud during patch orch

    When dcmanager creates a patch strategy, if a subcloud has an
    existing vim patch strategy, it will attempt to re-use
    that strategy during its patching phase, which may result in an
    error.

    This commit deletes the existing vim patch strategy in
    a subcloud, if it exists, so it can be re-created.
    If the strategy cannot be deleted, orchestration fails.

    Change-Id: Id35ef26ed3ddae6d71874fc6bac11df147f72323
    Closes-Bug: 1929221
    Signed-off-by: albailey <email address hidden>

commit 9e14c83f0162549a2a94cb8bc1e73dbc4f4d4887
Author: albailey <email address hidden>
Date: Tue Jun 1 14:37:14 2021 -0500

Adding activation retry to upgrade orchestration

    When performing an activation, the keystone endpoints may not
    be accessible in the subcloud due to the asyncronous way that
    cert-mon can trigger a restart of keystone.

    This would have occasionally resulted in the upgrade activation
    failing to be initiated, and orchestration needing to be invoked
    again to resume.

This 'hack' adds retries and sleeps to the initial
activation action.

    Change-Id: Ic757521dec7bdc248a51a70b5463caafe7927360
    Partial-Bug: 1927550
    Signed-off-by: albailey <email address hidden>

commit bb604c0a9b872efd65fa45f1e2269995818c6262
Author: Tee Ngo <email address hidden>
Date: Thu May 27 22:17:16 2021 -0400

Fix subcloud show --detail command related issues

    If the subcloud is offline, the command stalls and eventually returns
    the "ERROR (app)" output. If the subcloud is online, the oam_floating_ip
    info is excluded from the output when the subcloud id instead of subcloud
    name is specified.

This commit fixes both of the above issues.

    Closes-Bug: 1929893
    Change-Id: I995591368564539b0e6af185b1adba2db73e0e46
    Sign...

Reviewed:  https://review.opendev.org/c/starlingx/distcloud/+/796528
Committed: https://opendev.org/starlingx/distcloud/commit/4c5344f8765b372cb84d2b1181589c16db2ae6e4
Submitter: "Zuul (22348)"
Branch:    f/centos8

commit cb979811017bd193fc1f06e53bb7830fd3184859
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Wed Jun 9 11:11:27 2021 -0400

Format the IP addresses in payload before adding a subcloud
    
    The IPv6 addresses can be represented in multiple formats. As IP
    addresses are stored as text in database, ansible inventory and
    overrides, this commit converts the IP addresses in payload to
    standard text format of IPv6 address during adding a new subcloud.
    
    Tested with installing and bootstrapping  a new subcloud(RVMC
    configured) with the correct IPv6 address values, but with
    unrecommended upper case letters and '0'. The addresses are
    converted to standard format in database, ansible inventory and
    overrides files.
    
    Partial-Bug: 1931459
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: I6c26e749941f1ea2597f91886ad8f7da64521f0d

commit 2cf5d6d5cef0808c354f7575336aec34253993b3
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 20 14:19:24 2021 -0500

Delete existing vim strategy from subcloud during patch orch
    
    When dcmanager creates a patch strategy, if a subcloud has an
    existing vim patch strategy, it will attempt to re-use
    that strategy during its patching phase, which may result in an
    error.
    
    This commit deletes the existing vim patch strategy in
    a subcloud, if it exists, so it can be re-created.
    If the strategy cannot be deleted, orchestration fails.
    
    Change-Id: Id35ef26ed3ddae6d71874fc6bac11df147f72323
    Closes-Bug: 1929221
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 9e14c83f0162549a2a94cb8bc1e73dbc4f4d4887
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jun 1 14:37:14 2021 -0500

Adding activation retry to upgrade orchestration
    
    When performing an activation, the keystone endpoints may not
    be accessible in the subcloud due to the asyncronous way that
    cert-mon can trigger a restart of keystone.
    
    This would have occasionally resulted in the upgrade activation
    failing to be initiated, and orchestration needing to be invoked
    again to resume.
    
    This 'hack' adds retries and sleeps to the initial
    activation action.
    
    Change-Id: Ic757521dec7bdc248a51a70b5463caafe7927360
    Partial-Bug: 1927550
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit bb604c0a9b872efd65fa45f1e2269995818c6262
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu May 27 22:17:16 2021 -0400

Fix subcloud show --detail command related issues
    
    If the subcloud is offline, the command stalls and eventually returns
    the "ERROR (app)" output. If the subcloud is online, the oam_floating_ip
    info is excluded from the output when the subcloud id instead of subcloud
    name is specified.
    
    This commit fixes both of the above issues.
    
    Closes-Bug: 1929893
    Change-Id: I995591368564539b0e6af185b1adba2db73e0e46
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit f6a7ef1262d7e62e98cbcf1cefa45ce635b04e20
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Apr 22 09:01:41 2021 -0500

Cleanup Distributed Cloud Orchestration unit tests
    
    We have a DB setup that we can populate, so the subcloud DB values
    should not be mocked, and instead we should verify they are being
    properly set.
    
    Added unit tests for these states that had no tests:
     - kube upgrade 'updating patches'
     - kube upgrade 'deleting vim strategy'
     - kube upgrade 'upgrade simplex'
    
    Relocate a unit test that was in an old directory and was not
    being executed:
     - upgrade 'deleting load'
    
    Some tests were incorrectly changing the name of an object rather
    than its state attribute.
    
    hacking was incorrectly being specified directly within tox.ini
    rather than the test-requirements.txt file.
    
    FakeSubcloud is no longer being used anywhere.
    
    Story: 2008137
    Task: 42409
    Change-Id: I9d43a2b0f1923cbd7d469965968062503a58234d
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit d7986567d0111adeb175a9c106ad1f72b2e0c052
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu May 20 13:43:33 2021 -0400

DC dcorch log cleanup
    
    In order to increase retention period of dcorch logs, some info
    logs are moved to debug level.
    
    Change-Id: I8091b7fea6f0f8c3834802d03c438252529137ae
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1929111

commit 9f5b85c41c2d8aec2e550f187a8b40a7525033b0
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed May 19 14:12:07 2021 -0500

Specify the nodeset for zuul jobs
    
    The py2.7 jobs need to specify xenial
    The py3.6 jobs need to specify bionic
    The focal zuul nodes only have python 3.8 installed in them
    
    Zuul targets that invoke a generic python3 interpreter such
    as pep8 is not specified.
    
    The copyright date was updated in order to trigger
    the zuul jobs, as a no-delta type of change.
    
    Change-Id: If5352228a018f3147ae78172ae625066150c0a95
    Partial-Bug: 1928978
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 6b87e180298b19f5cfc441620cfdc546696ffb58
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed May 19 10:02:25 2021 -0400

Add support for k8s-upgrade-completed event notification
    
    This will be used by sysinv to notify dcmanager that the kubernetes
    upgrade has been completed so that it can trigger a kubernetes
    audit of all subclouds.
    
    Change-Id: I2ff48c939e808c3576a5739fa1dc37906e6a3a4e
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1928864
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792043

commit 077ab926e388d477a788e3cf306e2b7443c3927e
Author: albailey <Al.Bailey@windriver.com>
Date:   Fri May 14 13:48:23 2021 -0500

Updating some of the dcmanager and audit logs
    
    On startup of any process, the entire configuration was
    being logged, which was we were logging the entire
    configuration to the file (almost 200 lines per process).
    - Lowering that log to DEBUG.
    - Added an extra 'Starting...' log entry to help indicate
    when a process start/restart occurs.
    
    The update_subcloud_endpoint_status  log was not including
    enough information, as it is called several times in a row
    for each endpoint.
    
    Two audit logs that are created every 30 seconds have been reduced
    from info to debug.
    
    Closes-Bug: 1928335
    Change-Id: I7ada5abf87c2f28f5826c02345f8dd3197eae665
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 84dc98f5df97a8dba050a96e4cedace0e9419d8f
Author: Melissa Wang <melissa.wang@windriver.com>
Date:   Wed May 5 08:51:22 2021 -0400

Fix for subcloud upgrade activation failure
    
    This change adds exception-handling in case the haproxy is reset
    during upgrade activation so that it will retry the activation step
    
    Closes-Bug:1927550
    
    Change-Id: I678d50df3d8866518356d8da95b4742f0072c7fe
    Signed-off-by: Melissa Wang <melissa.wang@windriver.com>

commit 8d16d245504039b7e59b5d7a083b3b9d28b358e9
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 13 08:36:52 2021 -0500

Reset the force patch audit flag after it is triggered
    
    The force patch audit flag could be set, but never cleared.
    This would result in patch audits being run during every audit
    loop, as well as any derived audits such as software load.
    
    This change ensures that the force flag is cleared when a
    patch audit is invoked.
    
    The force patch audit flag is used by other audits such as the
    audit_load, so it needs to be reset after those evaluations.
    
    Change-Id: I833d157a616916a9ee37553b1f4cf3582caa8862
    Closes-Bug: 1928333
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 06d48e2ede3991d623089455e513918e07cf7ba4
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Thu May 6 13:51:32 2021 -0400

Extend the ansible timeout of the rehome playbook
    
    On a sub cloud that can no longer talk to its system controller,
    ansible cannot escalate the root privilege in the sub cloud until
    connecting to the LDAP server times out. This commit overrides the
    ansible default timeout(10s) to 180s of the re-home playbook, to
    prevent the re-home playbook failed by privilege escalation timeout
    before correcting the LDAP server address in a sub cloud.
    
    Tested:
    Re-homed a subcloud after powering off the previous system controller
    and deleting the static route to the system controller.
    
    Story: 2008774
    Task: 42408
    
    Change-Id: I65f9e0b7e242ef53a6fadcbab801ff39778b3d8d
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit b57f5ef65bea69db8563683c13d2299cccc35845
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed May 5 10:50:58 2021 -0400

Re-initialize RegionOne keystone client properly after subcloud add
    
    When a new subcloud is added, we expect IndexErrors while trying to
    fetch the subclouds identity auth URL. At this time, the RegionOne
    keystone client is also refreshed (by setting it to None). The
    next subcloud that tries to access the master keystone client
    cannot properly handle this and its availability is set to None for
    a short duration.
    This commit fixes the issue described above.
    
    Change-Id: I3139108b119aa6b6ba20277d829c14cc61e9f88c
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1927229

commit 17b5505d9ea9b149cf28236be3c1b4c263a89ffb
Author: Tao Liu <tao.liu@windriver.com>
Date:   Mon May 3 12:32:53 2021 -0400

Fix Sub clouds going offline due to auth failure
    
    This update contains the following changes that prevent subclouds
    going offline due to authentication failure:
    1. The os region client cache is cleared when a new keystone client
    is created. The os region client will be re-created using the new
    keystone session.
    2. When the user's access info (such as role id) is changed create
    new keystone client and os region clients. This could happen after
    system controller keystone role ids were synced to subclouds
    3. Remove get_admin_backup_session that was only required when
    upgrading to stx 4.0.
    4. Increase AVAIL_FAIL_COUNT_TO_ALARM to 2 as we don't want to alarm
    first failure since there are cases where we expect a transient
    failure in the subcloud (e.g. haproxy process restart to update
    certificates)
    
    Tested on DC-6:
    1. Adding 50 subclouds twice
    2. Soaking the fix over the weekend
    
    Closes-Bug: 1927007
    
    Signed-off-by: Tao Liu <tao.liu@windriver.com>
    Change-Id: I86fdc9a2f062409e704bdfac2119dc488123f7de

commit 8acd9699acec23c6ce4fe97bb1ea790d38032951
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed Apr 28 16:36:08 2021 -0400

Fix: Subcloud shows (patching) out-of-sync on the system controller
    
    The patching logic is modified to mark the subcloud as in-sync
    when the patches that are applied on the system controller
    have been committed on a subcloud.
    
    Change-Id: Ic56f0490cc0f7728ace721a2436d602a03ca1328
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1926635

commit 70a3475c73c851c4179d809d8e1b28ba46508683
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Apr 19 17:18:18 2021 -0400

Handling retry attempts of duplex subcloud upgrade strategy
    
    This commit supports retry attempts for duplex subcloud upgrade
    strategy when upgrade states are "data-migration",
    "data-migration-complete" or "data-migration-failed".
    
    Change-Id: I919228126a4960f712092c26b3abbf12e2de8ccb
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Story: 2008055
    Task: 42304

commit f5890513348e889824b61fda8867b14c7d0c4cde
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Apr 19 12:22:22 2021 -0400

Handle upgrade failure following unlock timeout
    
    In this commit:
    - Set the subcloud deploy status to DEPLOY_STATE_DATA_MIGRATION_FAILED
      if the subcloud host unlock/reboot times out so that the subcloud
      can be audited and handled correctly in pre_check upon upgrade
      retry.
    
    - Update pre_check to advance to activating step if deploy
      status is DEPLOY_STATE_DATA_MIGRATION_FAILED but the subcloud
      is online and passes health check.
    
    - Separate simplex and duplex pre_check logics for readability.
    
    - Correct a dcorch log that was not updated when the platform admin
      userid was changed from wrsroot to sysadmin.
    
    Closes-Bugs: 1925053
    Change-Id: Ie7f23a4f8de4e5dbcacfd7d625e15fe25d30e499
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit a9157c51d5562eb395815d4653d8e7d39b38920e
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Fri Apr 16 11:00:26 2021 -0400

Filter out skippable alarms in precheck
    
    Subcloud online checks are now skipped if the subcloud is
    already in the 'migrated' state. The precheck also skips
    upgrade alarm as well as the host lock alarm if upgrade
    has started.
    
    In addition, the commit includes the code to handle bmc_password
    with None value which the previous commit
    (69a74499884a8a73a3b705ce074f46202c4aa278) did not handle.
    
    Closes-Bug: 1924774
    Change-Id: Ifdfe1e44a34c2c2561c299a82d7178dce6063daf
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit 6c986880b7caa6687ffceaa1f3888d989756eb3e
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Apr 14 10:03:30 2021 -0500

Kube audit should report out-of-sync if a kube upgrade exists
    
    A subcloud can have all its kubernetes components upgraded, but
    there can still be a kubernetes upgrade object.  Audit will now report
    the existance of that object as meaning the subcloud is out of sync.
    
    Similar logic already exists for regular upgrades.
    
    Change-Id: If84c24bdd213c18fefde3752b1798f67aad560ae
    Story: 2008137
    Task: 42265
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit ef0123d0b48136edcef794ec03cad776100603e3
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Tue Apr 13 21:25:29 2021 -0400

Add restore related deploy states to auditable list
    
    In this commit, relevant restore related deploy states are
    added to auditable list so the subcloud can be set as offline
    while undergoing remote restore.
    
    Closes-Bug: 1923688
    Change-Id: Ib415327478c5ab45b61726f3ce2d6e88be0654cd
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit de96cf465b65661a60a8070f4293348fb962055d
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Apr 12 16:33:32 2021 -0400

Restore upgrade retry of offline simplex subcloud
    
    Code changes to support upgrade orchestration for duplex subclouds
    broke the upgrade retry functionality for offline simplex
    subclouds. This commit fixes it.
    
    Closes-Bug: 1923498
    Change-Id: I92d31df7ae8396293b302d357655470b447d143c
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 119dbaf87bb13a37e62b32d14dbd00776a1919db
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Feb 1 13:45:17 2021 -0500

Add a migrate option for rehoming a subcloud
    
    This commit adds a "migrate" option in dcmanager subcloud add API.
    With this option, the dcmanager can invoke a rehome_subcloud playbook
    to migrate a subcloud under another central cloud. The playbook will
    be installed under:
    /usr/share/ansible/stx-ansible/playbooks/rehome_subcloud.yml. The
    subcloud info will store in dcmanager database.
    
    The keystone password of "smapi" is required during the subcloud
    migration, so it is requested from keyring and added into the
    subcloud's overrides file.
    
    Tested:
    Happy path:
    On a system controller, trigger a successful run of the
    rehome_subcloud playbook with command:
    dcmanager subcloud add --migrate --bootstrap-address
    <bootstrap-address> --bootstrap-values bootstrap-values.yml
    In the boostrap-values.yml the systemcontroller_gateway_address should
    be set under the management subnet.
    The subcloud info is added into the dcmanager database. And its
    deploy_status is changed in the sequence: pre-rehome-> rehoming->
    complete
    
    Unhappy path:
    1 On a system controller, trigger a failed run of the rehome_subcloud
    playbook with the correct bootstrap values. The subcloud info is also
    stored in the dcmanager database. But the deploy_status is changed
    into rehome-failed in the end.
    2 The systemcontroller_gateway_address is not under the management
    subnet. The rehome_subcloud playbook is not triggered, and a notice
    indicates the problem is prompted.
    3 Create an error during creating the intermediate ca cert in
    cert-manager, subcloud's deploy status changes to "rehome-prep-failed"
    in the end.
    
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/780101
    Story: 2008774
    Task: 42150
    Change-Id: I758e102d386a63c7ed318bcc91d66e11d3f56299
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 69a74499884a8a73a3b705ce074f46202c4aa278
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Apr 8 14:40:30 2021 -0400

Encode bmc_password received from the subcloud host
    
    If the subcloud is configured to store bmc secret in its
    barbican database, ensure the password is base64 encoded
    before passing the install data to rvmc.
    
    Closes-Bug: 1923073
    Change-Id: I9639c8e09acb2d78013932d4123511b354bb04fb
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 8dcb56bb493078f34fec3c4f07786112cfbfacdb
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Mar 19 17:18:29 2021 -0400

Reduce keystone load for offline subclouds
    
    When a large number of subclouds are offline (e.g. powered down),
    the dcmanager-audit generates significant load on the keystone
    services in the System Controller when it periodically (roughly
    once per minute) creates a keystone token and then retrieves the
    endpoints. This causes CPU usage to spike. In order to address
    the issue, RegionOne client is cached in EndpointCache.
    
    Change-Id: I17db5cd0107457f07fe14febfec23da5290153de
    Story: 2007267
    Task: 42082
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 4a49932841abd3ca4f70eca500b37e60ba3ed746
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon Apr 5 15:01:53 2021 -0500

Only install yamllint in py3 env
    
    A new yamllint was released which dropped py2 support.
    Zuul servers that have updated their caches will attempt
    to install that version, as it is not clamped by upper constraints.
    
    This also attempts to not use site packages.
    
    Partial-Bug: 1922590
    
    Change-Id: I8b2776f4c12447a66743a8b2626aa914969dcd2a
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit b0b3cfe07b591648a12b669e36e6cc218d5328c3
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Apr 1 15:38:30 2021 -0400

Add API retries in finishing firmware update
    
    In firmware update orchestration, after the subcloud is unlocked,
    the runtime manifests on the subcloud may take some time to complete
    causing the sysinv API calls to query the subcloud to fail.
    This update is to add retry attempts during the finishing firmware
    update strategy step.
    
    Closes-Bug: 1922281
    
    Change-Id: I678eaf701d9c0cf40907fb8c378a5a2c531a88ae
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit d5fbf8913f0990f5d0e3b99edb79a8d8da801b12
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Mar 12 13:43:49 2021 -0500

Objectify firmware audit data
    
    Recently merged commit broke firmware audit and caused
    firmware_sync_status to be unknown even after the
    device image was uploaded on system controller. This
    commit fixes firmware audit by objectify the firmware
    audit data similar to patch and kubernetes audit data.
    
    Change-Id: Ibb53641b9ead78767523612a1cff8c8993c874dd
    Story: 2007267
    Task: 42048
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 36497a9bf518c52338e5f46d2f541124d742a8c2
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Mar 10 09:05:19 2021 -0500

Add unit tests for subcloud restore
    
    Update API and manager unit tests to include subcloud restore
    tests.
    
    Story: 2008573
    Task: 42028
    Change-Id: I0e16f9914fb5a4d0b73ea9837a75931e2a4432d6
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 7625b448de19d0a66ed3c98793e254671404e801
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Mar 8 16:51:09 2021 -0500

Handle in-progress subcloud operations after dcmanager-manager restarts
    
    When a subcloud is added it can go through several phases - install,
    bootstrap and deploy. If the dcmanager-manager process is restarted
    while the subcloud is being added, the DB would never be updated
    and subsequent phases are not completed. This commit fixes this
    issue by making dcmanager audit for subclouds in transitory states
    on startup and update the deploy state to the appropriate failure
    state.
    
    Story: 2007267
    Task: 42026
    Change-Id: I90a57cef89f1cc04a07fa6482ff2c29edf7be965
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 1348af4772414959561e40faa5d5a97e3cea11ce
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Mar 10 13:28:13 2021 -0500

Update dcmanager API doc for subcloud restore
    
    Document the dcmanager subcloud restore API.
    
    Story: 2008573
    Task: 42034
    Change-Id: I38505518b367b275a95f040cee67e6a183cf9606
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit c4a208325c6dd8403c9a93b74c50f1e7801097b0
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Wed Mar 10 10:19:51 2021 -0600

Remove existing dc-vault load data during import
    
    During a distributed cloud upgrade, an admin may need to delete and
    import a new load. If the load-delete command is issued without the
    --os-region-name SystemController parameter the load will be deleted
    from sysinv but will still be present in the dc-vault.
    
    To resolve this we will always remove any existing dc-vault data during
    a load import.
    
    Closes-Bug: 1918451
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>
    Change-Id: I143e2c084205c105f084d96714e8ebef94dbd130

commit 8e26a6731ae97b05f399dc52709b50eaa59054b8
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Mar 2 13:25:14 2021 -0600

add support for event notifications to dcmanager
    
    Add support for generic event notifications to dcmanager.  Initially
    this will be used by sysinv to notify dcmanager that the platform
    upgrade has been completed so that it can trigger a load audit of
    all subclouds
    
    Story: 2007267
    Task: 41968
    Change-Id: I9edb00a385ee5f617ac4415aa7e4675f52e28139
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>

commit b6638ce9a1055e5131a494d70897cbf37ce2a8c1
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Nov 17 15:47:58 2020 -0600

Distributed Cloud Kubernetes Upgrade Orchestration
    
    This commit introduces DC Kubernetes upgrade orchestration.
    
    - Added a new kubernetes endpoint for DC audit
      The audit compares that the active-target-version in the subcloud for
     kubernetes matches the system controller.
    - Wiring the kubernetes strategy to the kube orch thread
    - Add protections to orchestrator shutdown for when 'start' encounters
     a problem.
    - Added 'kubernetes' as a valid strategy type for:
       create / abort / delete / show
    - Update the endpoint flag file name to trigger updating the new
     endpoint
    - Added a kubernetes orch thread similar to sw upgrade orch thread
     with these states:
      - updating kube patches
      - creating vim patch strategy
      - applying vim patch strategy
      - deleting vim patch strategy
      - creating vim kube upgrade strategy
      - applying vim kube upgrade strategy
    
    Patch vim orchestration steps will be skipped if there are no patches
    that need to be applied in the subcloud.
    
    Change-Id: Ic508ed1a504dba177c100a9caf783e87bc10193a
    Story: 2008137
    Task: 41177
    Depends-On: https://review.opendev.org/c/starlingx/nfv/+/767421
    Signed-off-by: albailey <al.bailey@windriver.com>

commit 1602fc70b271c8986c0ba414300a9b27ff5f2e9d
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Fri Feb 26 08:57:11 2021 -0600

Support background runtime manifests during upgrade-activate
    
    Update dcmanager orchestrator to accept the new activating-hosts state.
    
    Story: 2008055
    Task: 41947
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>
    Change-Id: I032d82ed489b1ae0652429f91adfa5fa43fc3d3f

commit af14a6f65cfa13746ef9bed29a59e9640b4d1092
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Tue Feb 23 00:09:57 2021 -0500

DCManager update for Redfish subcloud restore
    
    In this commit, dcmanager api and manager are updated to
    support the restore of a Redfish capable subcloud from
    backup data.
    
    Unit tests will be added in a separate commit.
    
    Tests:
      - Restore request without install (not yet supported)
      - Restore request for a subcloud that is currently in 'managed'
        state
      - Load is not in dc-vault
      - Mandatory restore value (backup_filename) is not present
      - Specified backup file cannot be found
      - Subcloud restore with backup data on the target (i.e.
        backup tarball is under /opt/platform-backup)
      - Subcloud restore with backup data on the system controller
        (i.e. on_box_data = false)
      - Simulate install failure
      - Simulate check target failure
      - Batch subcloud restore
      - Subcloud restore retry
      - Subcloud restore with patches
    
    Task: 41725
    Story: 2008573
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/777046
    Change-Id: I8134c535e39231837727811475b0f01b2ccddb63
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 69b815af7d1e20eebc6fef2fe4b514a3355359ad
Author: Adriano Oliveira <adriano.oliveira@windriver.com>
Date:   Thu Feb 11 21:02:41 2021 -0500

Add orchestrator unlock retry logic
    
    During distributed cloud orchestrator upgrade, a SR-IOV VFs
    configuration change may take several minutes to apply because the
    runtime manifest may have delayed for several minutes.
    While not applied, if an unlock is attempted, it will fail because
    sysinv prevent it to happen since DB was not updated yet.
    Code has been added to detect host unlock failure and implement a
    retry logic that can take up to 20 minutes.
    
    Partial-Bug: 1914836
    Signed-off-by: Adriano Oliveira <adriano.oliveira@windriver.com>
    Change-Id: I51b1eba8cf7cba1368e159f9dbf0b4d6ee858285

commit c446fb2f2ff58dc3f4c4708ff5aae259d75ff4d1
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Feb 16 17:11:58 2021 -0600

dcmanager update availability on first failed audit
    
    In order to improve the response time in the face of failures,
    change the dcmanager aduit code to update the availability
    (offline/online) on the first failed audit rather than waiting
    for a second failure.
    
    This does mean that an intermittent failure will result in
    dcmanager treating the subcloud as "offline".
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: If71d51e82a0147ba38153bf06a6fa8f54d38ae80

commit 0a6ce1a956e6d9474c69a26d34ec733716b821b4
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Thu Feb 25 10:05:56 2021 -0600

Trigger all sub-audits when subcloud goes from offline to online
    
    When we detect a subcloud going from offline to online, we can't
    assume anything about it anymore so let's trigger all the sub-audits
    so that on the next audit we'll know whether it's in sync or not.
    
    (I also re-raised a couple of exceptions to get better error
    handling.  Shouldn't ever hit this but you never know...)
    
    Closes-Bug: 1916784
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ie252272ff75077e21a9942f136ebfbd95420c175

commit 3b2fb67cfe1556fb468363c5196056d54c8aad6b
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Feb 22 13:18:24 2021 -0600

trigger subcloud audit when subcloud becomes managed
    
    When we change a subcloud from unmanaged to managed, trigger all of
    the various subcloud audits for that subcloud.
    
    Previously we triggered them for *all* subclouds, but now we have
    the ability to do it for just one.
    
    Also change it to only trigger the audits if we successfully change
    the subcloud to "managed".  No point in triggering them if we
    weren't successful or if it was already managed.
    
    Story: 2007267
    Task: 41889
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: I4494c80080dacb59d330162035bd6b595d8026d2

commit 0b38bccc5f13ed4bf661d7b47a799c2eb2bbe1e2
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Feb 16 11:04:03 2021 -0600

add API to run all subcloud audits for one subcloud
    
    Add a SubcloudAuditManager API to trigger all of the subcloud audits
    for a specific subcloud.  This would be useful when adding a new
    subcloud.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: I015c7c5762a0fca1f08ff6e1aa92ca7a738ddb82

commit 843545184a89c1ae4c4907c7b8169a9eb60e29c7
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Fri Feb 12 12:23:48 2021 -0600

Trigger audits on dcmanager-audit startup
    
    If we swact controllers or patch dcmanager-audit, the audit processes
    can potentially be killed while there are audits in progress.
    
    Rather than wait for the full timeout, we trigger an "audit fixup"
    on dcmanager-audit startup, and also trigger all of the sub-audits
    in case we were doing any of them when the audit processes were
    killed.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ic069b6d10cae77613571faaba10869367182c118

commit fe05f2ea9bb6eba394229ad019b23f54eac31b69
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Tue Jan 26 17:38:17 2021 -0600

fix up stale subcloud audits
    
    If something happens and an audit is interrupted after it was
    started but before it completed, the timestamps will be messed up
    in the DB and a new audit won't be triggered because the audit
    manager will think that there's an audit already in progress.
    
    Add a new check that looks for subclouds where the audit has been
    "in progress" for long enough that we can be pretty sure something
    has gone wrong.  Once we find them, reset the audit finish time
    so that a new audit will be triggered at the next audit start time.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: I390a99ba2bb06915b60fcfb8865096e1ede36cd3

commit c71703128e99bdd47532524716684908d2a734ff
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Nov 30 17:46:11 2020 -0600

Move subcloud audit to new worker processes
    
    In order to allow for faster subcloud audits, introduce a new
    "audit-worker" process.  By default there will be four worker
    processes to perform the actual subcloud audits.
    
    The main audit process will scan the DB for subclouds which need
    auditing based on the new audit start/end timestamps.  It will
    then send out RPC messages to the "audit-worker" processes to
    request an audit for a list of subcloud IDs.
    
    When the "audit-worker" process receives the "audit_subclouds" RPC
    message, it loops over each of the specified subclouds.  For each
    subcloud it updates the DB to indicate that the audit is starting
    for that subcloud, then basically does the exact same audit that
    is currently done in the audit process, then updates the DB to
    indicate that the audit has been completed for that subcloud.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ifb3dd363fd337d24f2c3f7aaa3549624fffaceca

commit 8417ce7737a9798c0938a6e2d378db2961cb4636
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Nov 30 17:38:07 2020 -0600

Add subcloud audit DB table
    
    In order to allow for subcloud audits to be handled by multiple
    processes, add a new DB table to track subcloud audit information
    including start and finish times for the various audits as well as
    whether a special audit has been requested.
    
    On DB upgrade we populate the new table with entries for each
    non-deleted subcloud.
    
    The subcloud creation code is modified to also create a corresponding
    subcloud_audit entry.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Id2f2b49eb5b8b1f20b1c128bce4811479bcb96b9

commit a6cb0594193ce9cec4934a5b1b588b58d7f36082
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Sep 28 17:57:47 2020 -0400

Upgrade Orchestration for Standard subclouds
    
    This commit enhances upgrade orchestration to support Standard
    subclouds.
    
    Strategy Steps:
    - Pre-checks to verify subcloud is ready for an upgrade
      and to recover from previous failures
    - Install release N+1 license in subcloud
    - Download/import the release N+1 load in subcloud
    - Update and apply N+1 patches to release N+1 load in
      subcloud
    - Finishing patch strategy
    - Start duplex upgrade
    - Lock controller-1
    - Upgrade and re-install controller-1
    - Unlock controller-1
    - Swact to controller-1
    - Create VIM upgrade strategy
    - Apply VIM upgrade strategy
    - Swact to controller-0
    - Activating Upgrade
    - Completing Upgrade
    - Delete release N load
    
    Change-Id: I6fb60912ec1e90baa593cf98e502f4c91f599f42
    Story: 2008055
    Task: 41105
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit aab68918c9f307330692fa2cbff26a558566f066
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Wed Dec 16 14:36:53 2020 -0600

cleanup audit interval constants
    
    This converts the hardcoded patch interval to make use of an
    already-defined but unused config option, and tweaks the default
    audit interval.
    
    Story: 2007267
    Task: 41336
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
    Change-Id: Ie6f023d0e46accdb79bd854e9833158e0a33e4b3

commit 527a48513a2534ed73755f758e01992414cabe21
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Fri Jan 22 14:32:18 2021 -0600

Turn off the legacy pip resolver for distcloud
    
    The requirements that had conflicts have been updated.
    The local upper-constraints is no longer being used by tox
    since two constraints files with different values for the same
    requirements are not supported. To enable this change, some
    unnecessary code that required an older version of
    oslo.messaging is removed.
    
    Bugbear is also being disabled due to brokeness that can cause
    test failures.
    
    Change-Id: If007b7e0ad3c04b78b619542162267cc7d502460
    Story: 2004515
    Task: 41670
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 13092996ca4d3224354bd8dd39a9c50e256d5646
Author: John Kung <john.kung@windriver.com>
Date:   Thu Jan 14 09:20:02 2021 -0600

dcorch-engine endpoint type handling improvements
    
    Handle the endpoint type remove and add operations. This is
    invoked for optional applications such as stx-openstack.
    
    Cleanup unused SubCloudEngine class.
    
    Tests Performed with multiple subclouds:
      Bootstrap, Installation and Deployment
      Initial manage and sync
      Resource update and sync via proxy and audit
      Add and remove endpoint type
      Unmanage and manage subclouds
    
    Change-Id: I5c7033e7bed91a7bcc9b2c989daf333b76013b8c
    Story: 2007267
    Task: 41589
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 43138e6aa59bf057ae4c7d2cda623ed042fc2fc1
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Tue Jan 5 10:27:34 2021 -0500

Specified RVMC image tag for subcloud install
    
    In local docker registry, it is possible that there are multiple
    versions/tags of Redfish Virtual Media Controller(RVMC) image, caused
    by upgrade/downgrade platform for example. Currently, the subcloud
    RVMC install picks only the first returned image tag which may not
    compatible.
    
    This commit specifies the required RVMC image tag. And this image will
    be downloaded/pushed to the local docker registry as a static image
    after a fresh install or upgrade by the ansible playbook.
    
    Closes-Bug: 1908100
    Change-Id: Ic2a593ce81952e4b43f41ee9908b59ba51fe92ab
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 547b164fa3e72267c47c9ee344a75306d2f88460
Author: Tao Liu <tao.liu@windriver.com>
Date:   Mon Jan 4 10:25:53 2021 -0500

Orchestration Support for a Single Subcloud Group
    
    This update adds an option to create an orchestration strategy
    for a single subcloud group. It encompasses all types of DC
    orchestration - patch, upgrade, firmware.
    
    Test:
    Create patch-strategy with or without --group option
    Create upgrade-strategy with or without --group option
    Create fw-update-strategy with or without --group option
    
    Story: 2007267
    Task: 41524
    Signed-off-by: Tao Liu <tao.liu@windriver.com>
    Change-Id: Icda88d1bbfa537f3e9e399d7fabb48e2b2b6a7fa

commit ec8dba8a53b1beb0562bdd4228860f42e8d055e2
Author: Takamasa Takenaka <takamasa.takenaka@windriver.com>
Date:   Wed Dec 23 09:10:33 2020 -0300

Remove entries related to host-based snmp
    
    According to the host-based SNMP removal, remove community
    and trapdest related processes
    
    Story: 2008132
    Task: 41394
    Change-Id: Icad7f82d5e20149a49cbb1540486b882856ac189
    Depends-On: https://review.opendev.org/765381
    Signed-off-by: Takamasa Takenaka <takamasa.takenaka@windriver.com>

commit 181bd0c86d77dd783d0d1c82acdc0063732ed073
Author: John Kung <john.kung@windriver.com>
Date:   Tue Dec 8 10:43:50 2020 -0500

Enable multiple dcorch-engine worker processes
    
    Multiple worker processes are created for distributing the dcorch
    sync and audit work of the dcorch-engine on the active platform
    controller host.
    
    The database sync lock is introduced as a synchronization mechanism
    for dcorch-engine workers and stores subcloud, endpoint
    and action locks.
    
    The subcloud sync status table is introduced to track subcloud/endpoint
    sync request, audit status, and report status.  In-memory caching of
    subcloud objects and tracking info such as last audit and reported sync
    are removed and the dcorch DB is referenced instead.
    
    The GenericSyncManager and sync_threads are restructured and use
    thread pools for work to be performed on subclouds.  The infrastructure
    allows for sync_threads to be allocated to a dcorch-engine worker.
    In this commit, each worker process can handle each subcloud.
    
    Robustness audits ensures no stale sync_lock and dcorch-engine
    worker process file limits are increased.
    
    Tests Performed with multiple subclouds:
      Bootstrap, Installation and Deployment
      Initial manage and sync
      Resource update and sync via proxy and audit
      Unmanage and manage subclouds
      Verify sync_lock on controller swact and process termination
      Dead Office Recovery
    
    Co-Authored-By: Tao Liu <tao.liu@windriver.com>
    Co-Authored-By: Teresa Ho <teresa.ho@windriver.com>
    
    Change-Id: I179279dea5babaf57084b28adea86a03b330924a
    Story: 2007267
    Task: 41374
    Task: 38662
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 37959399c9d1e5a8915bf0df80ba51136a5e91f4
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Dec 9 12:13:20 2020 -0600

Fix bugs with assigning a group for a subcloud
    
    The subcloud CLI and API support passing a group by
    name or ID, however the code was only working if an ID was passed.
    
    Furthermore, it could bypass the validator and allow setting zero
    as the group id, even though there is no group with an ID of zero.
    
    This submission includes unit tests for updating the group by ID,
    name, and with a number of rejection tests.
    
    Change-Id: I53d20a090cfe031c55802104012cea1cb4ff6cb6
    Closes-Bug: 1907488
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 9f0c5426cc54775ec4f9d2f0ff89b709ae95b5fc
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Dec 8 16:07:51 2020 -0600

Fix tox failures from new resolver in pip
    
    New pip was released on Dec 3, 2020 which breaks downloads
    from pypi for some URLs.
    
    New virtualenv was released Dec 7 which includes the new pip.
    
    The new pip uses a different resolver which fails with many
    of the components installed by pip.  These components
    may need to be fixed by later commits, since the legacy
    resolver is being removed completely in early 2021.
    
    Change-Id: Ifcbd8965e2ef0b53dae0c878c2a0f2bd91b572c2
    Closes-Bug: 1907125
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 164e8be8dd972031547127b4c1a820dc27a4886c
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Thu Dec 3 10:12:49 2020 -0600

Fix pylint errors
    
    Fixing remaining pylint errors and removing from pylint disable list
    so they cannot be re-introduced in the future:
    E1102: not-callable
    E1120: no-value-for-parameter
    E1128: assignment-from-none
    
    Change-Id: I0f59fdcd0fd20e9d0a8042ba3f7f2a5a20a701db
    Story: 2007082
    Task: 41335
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit a5ad984ba2990a4cf42479c45bd4a246e4ac778e
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Nov 30 11:27:41 2020 -0500

Add a specific counter to handle activation failure
    
    We intend to retry activation 10 times if it fails(in a total of 45
    minutes) in the DC orchestration. As the audit and retry are sharing
    the same counter, if the activation fails after 10 minutes, the
    orchestrator will not re-attempt the activation.
    
    To handle the activation failure, this commit adds a specific counter
    to activate retry in case the activate fails after 10 minutes. After
    this commit, the orchestrator will wait up to 45 minutes if the
    activation is in progress, and the retries will be limited to either
    the audit reaches DEFAULT_MAX_QUERIES or itself reaches
    MAX_FAILED_RETRIES.
    
    Tested by activating an AIOSX subcloud by orchestration.
    
    Closes-Bug: 1904934
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
    Change-Id: Ic37c14ee316e02f690b55b082862a85fc54501bc

commit 326b8e33ddfc5854704a49b930f05229faf2124b
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Mon Nov 30 13:30:50 2020 -0600

Fix pylint E1101 (no-member) errors
    
    Fixing pylint E1101 errors and removing from pylint disable list
    so they cannot be re-introduced in the future.
    
    Some of the fixes included deleting or commenting out code that
    was no longer in use and was causing these errors.
    
    Change-Id: I5c5a7f9e90cc5cb9cf4510f578bb6e31cea1f5ef
    Story: 2007082
    Task: 41335
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit f7728125e1284edb064d471a92388b7dea074b80
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Nov 19 17:08:41 2020 -0500

Fix a logic flaw in orchestration pre-check
    
    If the subcloud is online but its deploy status is install-failed
    and the subcloud host is locked, the upgrading simplex step must
    have failed early in the previous upgrade attempt. The pre-check
    should transition directly to upgrading simplex step in the retry.
    
    Closes-Bug: 1904938
    Change-Id: I73815939526310d04d7b50dc8b85d33b5d502d25
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit b466efdb7e7e9ae2e45350a8d4cbe322383cb47c
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Sep 24 17:51:02 2020 -0400

Disable ansible logging when invoked via subprocess
    
    In this commit:
      - Overwrite the default ansible log path in dcmanager.
        The output of ansible playbook executed by dcmanager
        is already directed to /var/log/dcmanager directory.
        The default ansible logging to the playbook user's
        home directory (/root in this case) is unnecessary
        and can exhaust rootfs over time.
      - Write playbook output to /var/log/dcmanager/ansible
        directory.
      - Set separate logrotate rules for ansible output.
    
    Tests:
      - deploy subclouds and verify that
        a) no ansible output is written to /root
        b) ansible output (remote install, bootstrap, deploy)
           is written to /var/log/dcmanager/ansible, one single
           log per subcloud
        c) each instance of ansible output in the log file
           follows the following format: playbook command with
           timestamp followed by the output of the executed
           playbook
        d) subcloud ansible output log is rotated with delayed
           compression to ensure logging continuity for the
           on-going ansible play
        e) rotation rules (size, count) are respected
      - orchestrate a subcloud upgrade and repeat the above
        verifications
    
    Closes-Bug: 1895607
    Change-Id: Idbf87e4472df26c844526328aeacd9e55325bf0c
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit e88f3ef6f504c8b72fb4a3f9650a7f71e8ae6fb9
Author: Lu Yao Chen <luyao.chen@windriver.com>
Date:   Mon Nov 9 15:03:23 2020 -0500

Flag secret to mask dcmanager logs
    
    Using oslo_config flag and setting secret=True on sensitive info
    tested on running controller-0 system, api, audit and engine
    logs are masked by redacting the info
    
    Partial-Bug: 1896116
    
    Change-Id: I201ef344902ace391f6fa028ce81fd3fe96b92f3
    Signed-off-by: Lu Yao Chen <luyao.chen@windriver.com>

commit 83155e5e9f94bc1ab2e137f9b025b64119283e6a
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Tue Nov 3 20:23:39 2020 -0600

Update unit tests to use database instead of mocking
    
    Updating the distributedcloud unit tests to use the database
    during unit testing instead mocking all database access.
    This improves code coverage and simplifies the unit tests.
    
    Also fixing a minor bug in SubcloudManager.add_subcloud that
    resulted in an incorrect deploy_status being returned to
    the caller of the method (uncovered by unit test changes).
    
    Change-Id: I68eb9b98d440ae0c5f6661b17a590e6d7b1ff414
    Story: 2007082
    Task: 41210
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 561b24b285bbe5d3e651b34bc4136bc206f42af3
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Thu Oct 29 11:33:31 2020 -0500

Remove distributedcloud dependency on sysinv
    
    The distributedcloud package uses a single constant from the
    sysinv package. Removing this dependency - use of sysinv
    should only be through the REST API or the client - no
    sysinv code should be imported.
    
    Also changing tox config to move the tox work directory
    out of the source tree.
    
    Change-Id: I9973f16ae66a7339d2df958b5f2afe047ee980d2
    Story: 2007267
    Task: 41176
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 12b81ff452ce79d6dc1547c0a315284309e95b7f
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Sep 1 08:02:57 2020 -0500

Delete subcloud VIM strategies when DC fpga strategy deleted
    
    As part of an fpga upgrade strategy in a subcloud by the VIM,
    that strategy needs to be deleted.  Previously it was being
    deleted as part of a the final steps of a successful orchestration,
    but there can be scenarios where that orchestration is incomplete,
    and therefore these VIM strategies need to be handled.
    
    A new DC orchestration cannot be created until the old one is deleted,
    so this change ensures fpga vim strategies are deleted if possible
    when the DC orchestration strategy is deleted.
    
    A bug in applying vim strategy has been fixed, and the section removed
    since a vim auth exception cannot be raised since a new vim client
    is constructed per loop.
    
    Unit tests to invoke the delete strategy have been added.
    
    Tox coverage now generates an html report.
    
    Change-Id: I4f3d7bc3550e12c430d1b1c0ce3dffb298fe31b4
    Closes-Bug: 1891521
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 797574962804762a562a088b08b114f929ed6512
Author: Don Penney <don.penney@windriver.com>
Date:   Tue Oct 13 09:45:52 2020 -0400

Setup distcloud logfile configuration
    
    This update creates and packages distcloud-specific logfile
    configuration for syslog and logrotate, moving it out of the central
    syslog-ng-config package.
    
    Change-Id: Ia6d38fe7d39ff68178441db8844c00232f7597f6
    Story: 2008251
    Task: 41099
    Depends-On: https://review.opendev.org/757947
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit c659a7f684ca06e8104a66e95047d61966c865c2
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Mon Aug 17 17:42:08 2020 -0400

Create an API to reinstall a subcloud
    
    This commit creates a REST API to reinstall a subcloud.
    
    In SubcloudController, the tasks includes:
    1. Read the subcloud info from dcmanager db, subclouds table.
    2. JSONify the data_install from db, generate the new install values.
    
    In SubcloudManager:
    1. Check the subcloud availability, software version before reinstall
    2. Check the image value, if image doesn't exists, update the image with
    iso in dc-vault
    3. Run the install command and apply command
    
    API format: PATCH /v1.0/subclouds/<subcloud>/reinstall
    -----
    Tests:
    -----
    Happy path:
    1. dcmanager subcloud add: a new subcloud with bootstrap values,
    deploy config and install values.
    2. dcmanager subcloud delete: an existing subcloud.
    3. dcmanager subcloud reconfig: an existing subcloud with deploy
    config.
    4. dcmanager subcloud reinstall: an existing offline subcloud, this
    subcloud has image path in data_install in db dcmanager. After the
    tasks of reinstall and bootstrap succeed, reconfig this subcloud with
    proper deploy config, this subcloud will be online.
    5. upload an image to dc-vault using: system --os-region-name
    SystemController load-import -a <bootimage.iso> <bootimage.sig>. Then
    using dcmanager subcloud reinstall to an existing offline subcloud,
    this subcloud has no image path in data_install in db dcmanager. After
    the tasks of reinstall and bootstrap succeed, reconfig this subcloud
    with proper deploy config, this subcloud will be online.
    
    Unhappy path:
    1. dcmanager subcloud reinstall: an existing online subcloud,
    reinstall fails.
    2. dcmanager subcloud reinstall: an existing subcloud without
    data_install value in db, reinstall fails.
    3. dcmanager subcloud reinstall: an existing subcloud with
    data_install in db, but missing mandatory install value, reinstall
    fails.
    4. dcmanager subcloud reinstall: an existing subcloud, but its sw
    version in db doesn't meet the sw version with system controllers',
    reinstall fails.
    5. dcmanager subcloud reinstall: an existing subcloud with
    data_install in db, but has no image path in data_install, and also
    has no right versioned image in dc-vault in system controller,
    reinstall fails.
    
    Story: 2007267
    Task: 40732
    Change-Id: I8be6d8d11e6b4ee02bbcca499ba8869ba76bffaa
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 19790859f060c35b28e18b482a2406729c9a5915
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Sep 24 09:48:09 2020 -0400

Use cached keystone client in add subcloud workflow
    
    Leverage keystone client caching work done for dcorch to reduce
    load to keystone during batch subcloud deployment.
    
    Tests:
      - Deploy 1000 subclouds in 50-subcloud increments in a simulated
        DC environment.
    
    Story: 2007267
    Task: 40955
    Change-Id: I716b202005b943a1f389d577ce035371d7ab938a
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit eb97f4c8b6b3c1cce6c162c76b5309751cb7a924
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Aug 24 13:19:14 2020 -0400

Move dcmanager orchestration to a separate process
    
    1) Remove DC manager orchestration from dcmanager-manager process
    2) Create dcmanager-orchestrator process and associated files
    3) Add new RPC calls for dcmanager-orchestrator process to notify
    dcmanager
    4) Create/update unit tests, to verify the implementation
    changes
    
    Story: 2007267
    Task: 40734
    Change-Id: Ibbbae77558a8a8fd95b636fa6c3aebb1dfefb514
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 64caf6de7c0ba23fa0f5c92facc19400a11a3c4e
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Mon Aug 24 14:12:18 2020 -0400

Import load metadata for SX upgrades
    
    For SX upgrades use the load-import-metadata API to import the load.
    
    Do not sync patches for loads that were imported with metadata only.
    
    Change-Id: If106711c976c1514419b120ba933662014a9bd7a
    Story: 2007403
    Task: 40807
    Depends-on: https://review.opendev.org/#/c/749413/
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>

commit 4c29aa748cc1d4b1540cbbfe8096d35855119a65
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Wed Aug 26 15:30:30 2020 -0400

Skip patching during SX data migration
    
    For SX upgrades we will skip applying the patches present in the
    upgrade_data. The N+1 patches are not applied after load-import so would
    not be present in the upgrade_data. With
    https://review.opendev.org/#/c/745355/ all N+1 patches applied to the
    system controller will be present after the subcloud install.
    
    Change-Id: I8f174211d8a4ddf4d84502d9c81736aaa22bc0c8
    Story: 2007403
    Task: 40759
    Depends-on: https://review.opendev.org/#/c/748288/
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>

commit 8a14160c5396a190a4dc75db7f8e65f81ddbd3c8
Author: John Kung <john.kung@windriver.com>
Date:   Tue Aug 25 09:05:58 2020 -0400

Update upgrades and patching get client to ensure token valid
    
    The client access for upgrades and patching are updated to
    get the client with cached keystone token on each invocation.
    
    This ensures that the cached keystone token expiry is checked
    each time, and if necessary, refreshed.
    
    Tests Performed:
    Distributed Cloud patch orchestration
    Distributed Cloud upgrades orchestration
    Verify by shortening and forcing token expiry on long running tasks
    
    Change-Id: I4f69c16e3a61bdbdfd717f135681b82e9d860425
    Story: 2007267
    Task: 40741
    Signed-off-by: John Kung <john.kung@windriver.com>

commit fa09faf22994b37ee90c30a1e68a9567c835e074
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon Aug 24 15:13:30 2020 -0500

Update DC subcloud API unit tests for proper validation
    
    The DC subcloud API tests were mocking away many of the validators,
    and were testing generic failure for most invalid inputs.
    
    Now the unit tests for subclouds API endpoint will check success
    and failure scenarios more accurately.
    
    Change-Id: Ic0fd4c1d9f3552601f95bf5031f94982985019c3
    Story: 2007267
    Task: 40800
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit b442f2afcdadb4e10893b66c085a9690005f5946
Author: John Kung <john.kung@windriver.com>
Date:   Wed Aug 19 12:06:28 2020 -0400

Enable subcloud install from image in dc vault
    
    The subcloud install is updated to allow reference to the
    vault load iso.
    
    When the image is not specified in the install values, or the
    image specified matches the vault load location, the image input iso
    is updated to reference the System Controller load vault.
    In such case, the bootimage iso is no longer duplicated, thus reduces
    the storage requirements.
    
    The -a' or '--active' flag, enables the upload of an active image:`
    system  --os-region-name SystemController  \
        load-import -a <bootimage.iso>  <bootimage.sig>
    
    Tests Performed:
    Import active image into dc vault
    Import active image into dc vault with new image
    Subcloud Install from image in dc vault
    
    Change-Id: I9faab0a5c9c7be7861ce2b4bc785cd44b9d4771a
    Story: 2007267
    Task: 40695
    Depends-On: https://review.opendev.org/#/c/746972
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 7f25a7141d08718c111aad54141b26d34555be19
Author: Bin Qian <bin.qian@windriver.com>
Date:   Fri Jul 24 21:22:23 2020 -0400

Trigger cert-mon audit from subcloud state changes
    
    When subcloud is managed, or becomes online, inform cert-mon via RPC
    call, so that cert-mon to perform endpoint certificate audit.
    
    Story: 2007347
    Task: 40442
    
    Depends-on: https://review.opendev.org/#/c/742468
    
    Change-Id: I733d2e22c2a4f7089b903b428025d87cee5123c4
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit ed2ffb4f6b280cdaf0cec1d5625ed5c88d5ebdfb
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Aug 12 08:43:01 2020 -0400

Delete N load as part of DC orchestration
    
    In this commit, a new step is added to remove the N load
    after the subcloud has been upgraded to N+1 release. The
    importing load step is also updated to remove an old load
    in the subcloud should one exist prior to importing the
    N+1 load.
    
    Story: 2007403
    Task: 40656
    Change-Id: If015b94354d47931c2ac9090b6c10c29ee051c3c
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit fa3e10c7e4667cfd70c9dbc069e1a55b7a15f8fb
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Aug 13 18:19:20 2020 -0400

Add subcloud system health check to pre_check
    
    Subcloud upgrade can only start if system health check passes,
    i.e. subcloud node(s) are all unlocked, enabled, patch and config
    current; all critical components are healthy and there are no
    management affecting alarms.
    
    The commit also includes the fix for a minor bug related to no data
    install value.
    
    Story: 2007403
    Task: 40665
    Change-Id: I6464a20ea82f5ad0d6ac5e2bdc93bbcb28e08b6c
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit a93baec6fe472de8da5fa01aae42153d4087c6b5
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Aug 19 21:07:24 2020 -0400

Add support for rd.net.timeout.ipv6dad to subcloud install
    
    The rd.net.timeout.ipv6dad boot option instructs dracut in the initrd
    to use the specified timeout for its IPv6 Duplicate Address Detection,
    rather than its default of 50 seconds. This allows the user to
    configure an increased timeout for a subcloud that may take longer for
    interface networking setup to be complete.
    
    Story: 2007267
    Task: 40712
    Change-Id: Ic7fe8eca975625f8634fff310de3d3550d7f5450
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit 3360a8d3cb142ceb35a3bb17f691baa8a60929d2
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Aug 13 13:59:17 2020 -0400

Fix system commands in dcorch-sysinv-api-proxy
    
    The request body for the system commands was not accessible
    when processing the response. The fix is to make a copy
    of the request and its body prior to processing the request.
    
    Closes-Bug: 1891546
    
    Change-Id: I2a6202311d0f9f6fbb91ae9581e8818c69c78587
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit cc43aa6205a4adfffb289556304602be64db52af
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu Aug 13 13:15:40 2020 -0400

Exception handling when subcloud not found
    
    When subcloud alarms are fetched for a subcloud name that doesn't
    exist, an exception is raised, but this exception is not generated
    correctly. This makes debugging the issue more difficult. A fix has
    been made to generate the error properly.
    
    Change-Id: I7f2efb94d3063980e1d7edc8aa6b9a19eea2f5e1
    Closes-Bug: 1884575
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 82fa2e9b2fa70712146de4ba563c91b74d31b2f4
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Aug 11 11:44:02 2020 -0500

Set subcloud firmware in-sync immediately after subcloud completes
    
    When a subcloud completed, it might show out-of-sync until
    the remaining subclouds were also orchestrated since that
    is when the audit is explicitly called.
    
    Now we update the status as each subcloud completes.
    
    Change-Id: I0694c7aacfb781aae1a7d22dd777fa75dff2bdfc
    Closes-Bug: 1890952
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 589ab369f398c8a8283a64d19b742fefa19a9f22
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Jul 31 11:20:20 2020 -0400

Fixes to support DC FPGA Orchestration
    
    When dcmanager-audit is restarted, it ensures that all endpoints
    exist for all subclouds. If a subcloud without the endpoint is
    encountered during startup, it adds an entry for it without
    requiring the subcloud to be re-added. A flag is also added
    on a replicated filesystem to avoid these checks from
    being executed more than once on the system controller.
    
    Change-Id: I049404c6a3e881623f3ff905a859a94f03bcf8e1
    Story: 2007875
    Task: 40471
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 663fcf0f8868c57540e682d47b93aab3d7a0c12c
Author: albailey <Al.Bailey@windriver.com>
Date:   Fri Jul 31 16:12:46 2020 -0500

Fixes for DC firmware orchestration related to labels and upload
    
    The applied image labels on the system controller should be the labels
    used when applying the image on the subcloud.
    
    The stage that removes images now uses all the applied labels
    for the image when removing them.
    
    The code that converts image fields for upload was passing an integer
    for the revoke_key_id.  It now converts to a string to prevent an
    encode exception.
    
    This commit also enables some unit tests for DC firmware
    orchestration that were being skipped.
    
    Closes-Bug: 1890355
    Closes-Bug: 1890340
    Closes-Bug: 1890324
    Change-Id: I1a730b01c0bbaeb5f3e1f6ecfb19f649911da453
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 5aab8597d40856f115e238f18a283d2964588e39
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu Aug 6 16:06:01 2020 -0400

Trigger firmware audit immediately once subcloud is managed
    
    When a subcloud is managed, patch audit and firmware audit
    are triggered. However, the patch audit on completion resets the
    force_firmware_audit variable, which prevents the firmware audit
    from immediately being triggered and keeps the subcloud out-of-sync
    until the next firmware audit cycle (Once in 1 hour). Thus, a fix
    is added to address this issue.
    
    Change-Id: Ie1aa2c7b4cb706d32b17f33bffa17d1004721887
    Closes-Bug: 1890294
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 749f7dc3d96f2842cd80b59d8022ae433e376494
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Fri Jul 24 10:52:23 2020 -0400

Add REST API timeout to patching and sysinv clients
    
    The REST API calls from the distributed cloud should timeout to
    prevent process to hang forever.
    
    Change-Id: Iea7f05018c348a817758f6bc920489127242ed11
    Closes-Bug: 1886037
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 537d2a9749337fdec501d834f1a84021e5bbd56a
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Fri Jul 31 09:23:38 2020 -0400

Extend default periodic interval for resource sync audit
    
    This commit extends the periodic time interval for resource sync audit to
    prevent the elapsed time of the dcorch audit exceeding the previous 10
    minute period along with the increment of a large number of subclouds.
    
    Change-Id: I96c93aef28c6b6359a03f652c6bb4618220a28db
    Closes-Bug: 1889506
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 8dcf8967da9f856e49b5038e5372d6b55d4326c6
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jul 30 17:57:14 2020 -0500

Fix issues in DC firmware orchestration
    
    Labels were not being properly passed in to the apply image API.
    
    If removing labels, a syntax error could cause the importing
    firmware state to fail.
    
    Change-Id: Ia5c378f373668c9b9e82b0b422af15deddd42849
    Story: 2007875
    Task: 40468
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 2b953fd0c7d8d4973d179df34d5497492a56c9f7
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Thu Jul 30 19:07:31 2020 -0400

Add --force option when managing a subcloud
    
    When managing a subcloud, allow a --force option to be
    specified. The force option will allow an offline subcloud
    to be managed.
    
    This is necessary in cases where the user has (mistakenly)
    unmanaged a subcloud that is offline and the subcloud
    cannot be recovered while it is unmanaged. One example is
    a failure during DC upgrade orchestration that leaves the
    subcloud offline. To rerun upgrade orchestration and
    recover the subcloud, the subcloud must be managed.
    
    Story: 2007403
    Task: 40466
    Change-Id: I864de63b40b1065ffb1c6a025b5b221b793fbaf7
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 74027ac73986dc9eb8806ba3ebd836c6ba5a5194
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jul 8 15:44:13 2020 -0500

Add fw-update strategy type support for DC orch
    
    Firmware update states are:
     - 'importing firmware'
     - 'creating vim strategy'
     - 'applying vim strategy'
     - 'finishing fw update'
    
    Change-Id: I6f39f45a8b0f9d6b1ed8d33abb49715bb11a8496
    Story: 2007875
    Task: 40421
    Co-Authored-By: Jessica Castelino <jessica.castelino@windriver.com>
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 54ef00192d59b43c104d94229b4d66ea0a11b27e
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Thu Jul 23 08:06:07 2020 -0400

Add master reauthentication for identity data sync
    
    This fixes Issue #4 from Bug 1870999 where some subclouds failed to sync
    an identity resource change.  This was due to unauthorized request error
    on the master cloud due to expired token.  The current error handling
    only covers reauthentication of subcloud dbs client.
    
    In addition, the number of sync tries is increased from 2 to 3 to
    accommodate the case were data sync fails authentication on master and
    subcloud.
    
    Also fix exception in dcdbsync on subcloud due to uninitialized variable
    when handling change to project resource from dcorch audit on system
    controller.
    
    Change-Id: Ife8d86481a64ed841f5231a6df055028ee6597b0
    Partial-Bug: 1870999
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit 598178e36ed6039d2064a81cbd3adde3a51edddd
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Jul 16 14:10:25 2020 -0400

Support device image commands in dcorch-api-proxy
    
    The dcorch-api-proxy is updated to support the
    device image upload, apply, remove and delete operations.
    
    Story: 2007875
    Task: 40371
    Depends-On: https://review.opendev.org/#/c/742457
    
    Change-Id: Id5506226297e723f19b7d03ca18cc47ddfa8af40
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit 6fd2b44e750cec2949c7adff737b9c89dae7270f
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jul 21 17:21:21 2020 -0400

Firmware audit
    
    As part of the dcmanager, there will be a periodic audit which
    scans a number of subclouds in parallel and checks whether the
    subcloud has all of the “applied” device images that it should
    have (based on the labels the images were applied against and
    the host labels in the subcloud), and whether all of the “applied”
    device images have been installed on the hosts that they should
    be. If either of these is not true, then the subcloud
    “firmware image sync status” is considered “out of sync”. This
    will result in the subcloud as a whole being considered “out
    of sync”.
    When a device image is “applied” in sysinv, dcmanager
    will be notified and will set the “firmware image sync status”
    to “unknown” for all subclouds since it does not know at this
    point which subcloud(s) the image needs to be created or applied
    or updated in. On the next audit, this sync status will be
    updated to “in sync” or “out of sync” as applicable.
    
    Change-Id: Ifbbbeaba0bcd2691332e23d288856b8417c9efa8
    Story: 2007875
    Task: 40420
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 9a4ad4921a56556eb47968257f80da49cb2e586b
Author: Andy Ning <andy.ning@windriver.com>
Date:   Fri Jul 24 14:19:34 2020 -0400

Fix subcloud slow initial sync caused by keystone reauthentication
    
    Under certain conditions subcloud keystone client reauthentication
    will fail during initial sync. It will keep on failing in reattempts,
    causing endpoints to go in-sync slow after the subcloud is managed.
    
    This is caused by the keystone reauthentication reusing the existing
    token that is already invalid. Once that existing token expires
    (an hour at maximum), the reauthentication will succeed and endpoints
    go in-sync.
    
    This update fixed this by recreating subcloud admin session (and
    keystone, dcdbsync clients) by using username/password. With this
    the reauthentication will cover cases where subcloud admin session
    user changes (eg, from admin to dcmanager or vise versa).
    
    Change-Id: I8a49bf8c55e3538fc47b833ae648b667b5e9e9e5
    Closes-Bug: 1887849
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit 0620b421fd7999ab37747114c0a0f5ec8a098dd5
Author: John Kung <john.kung@windriver.com>
Date:   Tue Jul 28 09:28:22 2020 -0400

Update dcmanager upgrade orchestration client access
    
    During upgrade orchestration with long running operations, it is
    possible that the keystone token may expire and result in
    unauthorized http errors.
    
    In dcmanager upgrades orchestration, the client usage is updated to
    retrieve the (cached) keystone client to ensure the checks for the
    token expiry are up to date.
    
    The unauthorized http exception handling is made explicit for the long
    running client operations.
    
    Change-Id: I281ecc51b1a4349ac865f0c4f6ea8244b7cb1fcc
    Story: 2007403
    Task: 40452
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 49f7ad12fa61430842d7da8663ec9dc8c7378f57
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Jul 22 12:25:10 2020 -0400

Update DC ICA expiry duration to 1 year
    
    The current 10 years expiry duration is incorrect. Change it to 1 year.
    
    Change-Id: Ia9cb8c104c1bb2b1028b29ea1cb39c8aea9d235a
    Closes-Bug: 1888521
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit 73998010b58073a15bc99bf58928376433e10937
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Jul 13 21:49:57 2020 -0400

Add recovery code for failures during each upgrade state
    
    In this commit:
      - Pre check for online case is updated. If the subcloud
        deploy status is 'migrated', advance directly to activating
        upgrade step. Otherwise, start from the first step
        (installing license).
      - The subcloud deploy status is set to 'migrated' at the end
        of migrating data step.
      - The subcloud deploy status is set to 'complete' at the end
        of completing upgrade step.
      - Skip data migration if deploy status is either 'migrated'
        or 'complete'.
      - Reduce the number of retries for failed activation from 45
        to 10. The overall wait time is still 45 minutes.
      - Improve error messages to user.
      - Add another log to give clarity to the time it takes the
        subcloud to respond to load import request.
      - Various small fixes.
    
    Tests:
      - Load import failure - not enough disk space
      - Load import failure - bad load file
      - Upgrade start failure - sysadmin directory too large
      - Controller lock failure - processing failure
      - Upgrade simplex failure - incorrect bmc password
      - Upgrade simplex failure - network glitch
      - Data migration failure - network glitch
      - Upgrade activation failure - processing failure
      - Upgrade completing failure - processing failure
    
    Story: 2007403
    Task: 40351
    Change-Id: Idf0bf8009122018e4597a7f1dede58658d0a064b
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit f0dd6b9101ec28ef76faa0bf4b422345cf1d7bbc
Author: John Kung <john.kung@windriver.com>
Date:   Mon Jul 20 17:29:35 2020 -0400

Add prechecks for data_install and scratch filesystem
    
    As part of precheck state checks, check data_install values and
    subcloud host scratch filesystem size.
    
    The data_install values are semantically checked on subcloud update,
    so the precheck verifies its presence.
    
    Tests Performed:
    Verified data_install empty check
    Verified subcloud scratch host filesystem check when online
    Verified pre_check stage continues to next stage on success
    tox unit tests updated and passed
    
    Change-Id: I972309f08d648cad6719b9a340a876258f6f912a
    Story: 2007403
    Task: 40396
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 43492c97db804e6a88053b1abd63d4ddf35749e4
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Fri Jul 17 16:55:51 2020 -0400

Add --force option to strategy create API
    
    The new option will allow the user to retry the upgrade of the
    specified subcloud that previously failed during remote install
    or data migration step.
    
    Story: 2007403
    Task: 40387
    Change-Id: Ib17857340c85892ff962bfcf3f9793c4e65bbef2
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit dc6d66c969c9236890013f2a029c8e65d422bac5
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Jul 8 17:06:16 2020 -0400

Add RestAPI to update subcloud sync-status
    
    Currently set restriction to only allow updating endpoint dc-cert
    Updating other endpoint status can be enabled when there is use case.
    
    Passed TCs:
    Test with testing client to
    1. successfully update dc-cert to in-sync, out-of-sync and unknown
    2. updating endpoint other than dc-cert is rejected
    3. updating dc-cert status to word other than above mentioned is
       rejected.
    
    Change-Id: I77df581c9d1d461f72948499e9410f22819b1030
    Story: 2007347
    Task: 40300
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit 6a16d05061e23c87c53b2ca6e89a2b383db8f850
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Wed Jul 15 23:53:10 2020 -0500

Fix dc-cert sync status handling for unmanaged subcloud
    
    When a subcloud is unmanaged and then managed again, it will
    never go back in sync. This is because all sync_statuses are
    set to unknown when the unmanage is done. However, the
    dc-cert_sync_status is only set to in-sync when the subcloud
    transitions from offline to online. So in this case the
    dc-cert_sync_status stays as unknown.
    
    The fix is to avoid setting the dc-cert sync status to unknown
    when a subcloud is unmanaged. The dc-cert sync status will
    still being audited for unmanaged subclouds so there is no
    need to do this.
    
    Change-Id: I7a53c11888b8866c6077679265d69c36faf4e264
    Closes-Bug: 1887733
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 6408de3d7e36c8837cb7b37c670455cb80efca61
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue Jul 14 09:48:20 2020 -0400

Reduce the patch audit interval
    
    Currently the patch audit assumes it is waking up
    every 20s so it will wait through 300/20 audit cycles.
    Should the subcloud audit take more than 30 seconds,
    the patch audit only runs about every 13-14 minutes.
    
    This update stores a timestamp for the last patch audit
    and compare with the current time when deciding if an
    audit is required. In addition, it triggers the load
    audit when a patch audit is forced.
    
    Story: 2007267
    Task: 40342
    
    Change-Id: I29e096fd3f2c82f42263f737390b64452bb606c1
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 91e79f1d608489c39025695689343458e18da5c9
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jun 24 12:23:51 2020 -0500

Add a 'type' filter for update strategy API calls and DB filtering
    
    The CLI and manager classes can pass an additional 'type' filter so
    that upgrade and patch strategy commands are independent of each other.
    
    This allows the patch orchestration CLI to only show and have control
    over patch strategies. Horizon will also only show and manage patch
    strategies.
    
    Note: there can still only be one strategy in the DB at one time.
    When the api filter attribute for the update type is not passed to the
    URL, any strategy type can be queried.
    
    The 'create' and 'show' commands explicitly indicate when a mismatched
    strategy type is present.
    
    Change-Id: I40058b8cd5d09fa2b700f0a45d10a6b59b74778c
    Story: 2007403
    Task: 40348
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 4e4355458d0f680bf150032000e1ca9b84682e2d
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jun 17 11:05:01 2020 -0500

DC upgrade state machine states will select next state
    
    Rather than having the state machine states as an ordered list,
    this modification allows each state to specify the next state.
    
    This enhancement allows future features like compound states
    or a pre-check state to enable skipping over known states
    or alternative transitions.
    
    This adds a pre-check state. Only the success path has been tested
    so far.
    
    Change-Id: Ideae9885b6026cb441bcb6ac08204784b876f3a9
    Story: 2007403
    Task: 40347
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit b5e45f15fe697295b18c62c678c745b3d798cd77
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Jul 13 21:58:50 2020 -0400

Update resources purge logic
    
    The daily dcorch database purge cron job deletes resources that are
    no longer used by any orch jobs but as the subcloud_resource table is
    set to cascade delete it also removes subcloud resources that are
    still in use. This is then flagged by the next dcorch audit and the
    subcloud resources are re-created over the next audit cycle(s).
    
    In this commit, only purge resources that are not referenced by any
    orch jobs and subclouds.
    
    Closes-Bug: 1887430
    Change-Id: I7179a2fba21f296dd51660bb224f5d51aaf531d8
    Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>

commit 3941475e5f17998b6ddbc3426be46db65135ac98
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed Jun 24 13:50:11 2020 -0400

Subcloud-group based orchestration
    
    Whenever an orchestration strategy is created, it will ensure that
    application of patches or software upgrades would maintain the below
    ordering for the subclouds :
    1) All subclouds in default group first.
    2) Then all subclouds in the “first” group, then the “second”, etc…
    3) Never include subclouds from different groups in the same “step”
       of the strategy, to ensure they are not patched at the same time.
    4) If max_parallel_subclouds and subcloud_apply_type are provided while
       creating strategy, it will supercede the max_parallel_subclouds and
       update_apply_type values set for each subcloud group
    5) If max_parallel_subclouds and subcloud_apply_type are not provided,
       we must respect the max_parallel_subclouds and the serial vs.
       parallel choice set for each subcloud group.
    
    Change-Id: Ifc98e303f70716f3c7e528167658617a6455ca65
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Story: 2007403
    Task: 40164

commit 241ad4b31df13d0026c2c987e75a5e84b5d1d86d
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jul 7 12:20:38 2020 -0500

Allow audit for additional deploy status failed states
    
    Adding the deploy status failure states to the list of auditable
    criteria for subclouds.  This allows subclouds to be set to
    offline by the audit rather than skipping the audit for them.
    
    Change-Id: Ia4ce5b36c7e108afd8da98412ee4bc8c29c3a49c
    Closes-Bug: #1885817
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 4f5d0b4fca0d9494f3acfd32349848558a9c34eb
Author: Yuxing Jiang <yuxing.jiang@windriver.com>
Date:   Tue Jun 30 11:50:15 2020 -0400

Return master_id from get_resource_id for deleting SNMP resource
    
    This commit modified the get_resource_id method in SysinvSyncThread.
    The previous code only return community/ip_address which is for the
    creat service operation. By this change, the get_resource_id method
    in this class will return resource.master_id for deleting SNMP
    community and trapdest.
    
    Test:
      1 Add snmp trapdest and snmp community configuration from system
    controller, with command:
        system snmp-comm-add -c test1
        system snmp-trapdest-add -c test1 -i 10.11.2.1
      2 Wait the snmp community and trapdest configured in subcloud in
    subcloud controller:
        system snmp-comm-list
        system snmp-trapdest-list
      3 Delete the snmp community and trapdest in system controller:
        system snmp-trapdest-delete 10.11.2.1
        system snmp-comm-delete test1
      Expected: trapdest or community should be deleted from subcloud,
    result can also be observed  in dcorch DB and /var/log/dcorch.log.
    
    Change-Id: I8d3b40599f83263cade1799d5a1dae72d3bac2c5
    Closes-Bug: 1863045
    Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>

commit 4f1c5db8094fc82b8b1defb5e574cb3e54cd4035
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 30 14:08:43 2020 -0400

Sync only ‘ssl_ca’ and ‘openstack_ca’  certificates to subclouds
    
    The 'docker_registry', 'tpm_mode', and 'openstack' cert on the
    SystemController will have a CN and SAN with OAM IP Address, any
    related DNS Names for OAM IP Address and MGMT IP Address. If these
    certificate are synched to the subclouds, even if they are signed by
    a trusted CA, when any client connects to the subcloud OAM or MGMT
    IP, the certificate that comes back will not pass certificate
    validation because the certificate does not apply to the subcloud's
    IP Address. For this reason a fix is put in place to avoid only sync
    "ssl_ca" and "openstack_ca" certificates to subclouds.
    
    Change-Id: I17ac94d79beb04f559c46062dbe7826590fcdb06
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Closes-Bug: 1865643

commit 0d1799d810943d9d651ef919ca23f23f56c91d9d
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jun 30 13:44:30 2020 -0500

Increase the DC orch upgrade-activation timeout
    
    Testing on one subcloud took 19 minutes for the upgrade
    activation to complete, and so it timed out.
    
    The timeout has been increased from 15 minutes to 45 minutes.
    
    Change-Id: Ice69acbbe097ea038e2932edb44f59f024cbeefb
    Closes-Bug: 1885451
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit cfd225b75bdf355b5df7c0125078dd15c03584b5
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jun 25 09:28:31 2020 -0500

Adding a wait to the end of migrate_data state
    
    The ansible script triggers a reboot and the state needs to
    wait for the reboot to complete, before the state transitions
    to the next state.
    
    This adds a retry for the upgrade-activate step, and increases
    the total max time for that task to be 15 minutes
    
    This also fixes a logging bug where a failure in one subcloud
    causes excessive logs while other subclouds are being processed.
    
    Closes-Bug: 1885132
    Change-Id: Iabca2ecb0bacff7c5622c90722fcfb938a71a768
    Story: 2007403
    Task: 40172
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 8ecba753119da83e80f4933bd24dac311b5979f1
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Thu Jun 25 14:16:45 2020 -0400

Add timeouts to DC sysinv and patching clients
    
    If a subcloud is powered off or loses communication with the system
    controller while the subcloud audit is waiting for a response to a
    sysinv or patching request, the subcloud audit will hang forever.
    Add some failsafe 10 minute timeouts to ensure the audit can recover.
    
    Change-Id: I369225119f89ab5a2f05137aba955dda982d51c5
    Closes-Bug: 1884560
    Closes-Bug: 1870413
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit dd9c780663af60666e82d2441e242dd6ca1e72f9
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue Jun 23 17:02:19 2020 -0400

Add CLI/API to populate data_install
    
    The dcmanager subcloud update API is extended to
    allow populating/updating the data_install values required
    for remote install (upgrade to stx 4.0):
    
    In addition, the bmc-password is stored as base64encoded.
    
    Test cases:
    1. Add a subcloud without install-values and update the
       subcloud with install-values
    2. Update the subcloud install-values with new data
    3. Verify the bmc_password is stored as base64encoded
    4. Add a subcloud with install-values and verify the
       bmc_password is populated in rvmc.conf as base64encoded
    5. Subcloud update with the previously supported parameters
    
    Story: 2007403
    Task: 40149
    
    Change-Id: I78db37873547fde43f085a0093765f838d2e4200
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit a331e544fa16b8cee20c7cc392da1b34805e6c1e
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 23 11:31:16 2020 -0400

Distributed Cloud re-execution susceptible to old data
    
    Added an os_O_TRUNC flag to ensure that files are truncated
    before writing new content to it. This avoids previous content from
    being appended to the new file.
    
    Change-Id: I14438d4276376089c1c46a20626f251adcac09f9
    Closes-Bug: 1883113
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit e422388185102616b35946a6bd04d038e415380a
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jun 11 15:22:22 2020 -0500

Add thread loop breakout handling for DC upgrade states
    
    If a state machine handler has internal loops, it can check to
    see if the main thread is being stopped, and can break out early.
    
    The states performing this check are:
     - importing load (can take up to 30 minutes)
     - starting (can take up to 30 minutes)
     - lock controller (can take up to 10 minutes)
     - unlock controller (can take up to an hour)
     - activating (can take up to 10 minutes)
     - completing (can take up to 10 minutes)
    
    Added a loop-check for 'starting' state to wait for
    upgrade to move to 'started'.
    
    Added a loop-check for 'completing' state to wait for
    upgrade to be deleted.
    
    Change-Id: I2d4e77d1baf917738ddc96f3cbaa76249ba17fdf
    Story: 2007403
    Task: 40105
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 36dce2d5f072ba464e9cb1fe82e80bf14c6dc943
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Jun 10 13:26:55 2020 -0500

DC upgrade state handler improvements
    
    1) DC upgrade importing-load state should wait for 'imported'
    
    The importing_load state for software upgrade orchestration was
    moving to the next state after the load was 'importing' instead
    of 'imported'.  This caused the next state to fail since it required
    the load to be in the imported state.
    
    2) Use the strategy options object 'alarm-restriction-type' to
    determine if an 'upgrade start' should use a force to ignore alarms
    that are not management affecting.
    
    3) Add polling stage after upgrade activation to wait for the activity
    to complete before transitioning to the next state.
    
    4) Update the unit tests to mock timeout values so that
    running the unit tests is quicker.
    
    Change-Id: Ibaf77d2c33b633f915301344c7a7318a59fe3a9f
    Story: 2007403
    Task: 40040
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 1012dd28964b7416e24b3dd60b217988d137e640
Author: John Kung <john.kung@windriver.com>
Date:   Tue Jun 9 10:22:56 2020 -0400

Implement the state for Simplex Upgrade
    
    The Simplex Upgrade step upgrades the simplex subcloud to the load
    imported in dc-vault corresponding to the target software release.
    
    Required static data is captured at subcloud add and referenced for
    the simplex upgrade.
    
    Dynamic data is retrieved from the online subcloud; including the
    bmc credentials which are obtained via a synchronized service within
    project allowing for access to the bmc credentials.
    
    BMC password retrieval endpoint_cache integration
    with https://review.opendev.org/#/c/736004/
    
    Tests Performed:
    - Subcloud upgrade (install) with load corresponding
      to System Controller
    - Subcloud add persists install data
    - Simplex upgrade persists dynamic upgrade data from start of
      upgrade simplex
    - Obtain bmc password from online subcloud
    - Obtain bmc password from install data, in case the
      bmc is not configured
    
    Pending for subsequent commit:
    - tox unit tests
    
    Change-Id: Ie047139280a5780bfe47b9f9959f4c6781d6f3fa
    Story: 2007403
    Task: 40024
    Signed-off-by: John Kung <john.kung@windriver.com>

commit 94946613cdb51e1a7b9c08ad8344d120da0ed59f
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Fri Jun 12 13:14:57 2020 -0400

Implement simplex subcloud data migration state
    
    This commit implements the data migration state for subcloud upgrade
    orchestration.
    
    Story: 2007403
    Task: 40071
    Depends-On: https://review.opendev.org/#/c/734620/
    Depends-On: https://review.opendev.org/#/c/733902/
    Depends-On: https://review.opendev.org/#/c/733914/
    
    Change-Id: I5b38f31324ddb62b6760f897a1eb3cca5159e2ef
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit d10fef48217d69a45084dd8b81c30d3bbe68b3af
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue Jun 16 15:08:20 2020 -0400

Configure dcmanager user for endpoint_cache
    
    The following changes are made, in order to remove the
    dependencies to 'admin' user and use 'dcmanager' user to
    authenticate with services in the subclouds:
    . Add endpoint_cache section to both dcorch and dcmanager
    . Configure dcmanager user in the endpoint_cache section
    . Sync the dcmanager user to subclouds
    . Use 'dcmanager' user for subcloud authentication by default,
      and it falls back to 'admin' user if the authentication fails
     (This is required during software upgrade).
    
    Depends-On: https://review.opendev.org/#/c/735994/
    Partial-Bug: 1883758
    
    Change-Id: Ibc78abc86a7a825f83f2cac9fd54e4183e7ccd80
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit e8d916756d439cf63eed52f347a2a547fc5124cb
Author: Andy Ning <andy.ning@windriver.com>
Date:   Wed Jun 17 10:40:23 2020 -0400

Set up /var/run/dcmanager as dcmanager's default temp files location
    
    dcmanager call the k8s python client to perform a number of
    operations. The k8s python client creates temp files under /tmp and
    continues use these tmp files for the life-cycle of the processes.
    
    However systemd-tmpfiles-clean.service will run every day to clean up
    files in /tmp dir that are older than 10 days. If the k8s client code
    is not triggered for more than 10 days (thus its temp files are not
    accessed for more than 10 days), these temp files will be removed as
    part of the cleanup. Certain dcmanager operations then starts to fail
    with an error that the tmp file is no longer there.
    
    This is a known issue of kubernetes python client:
    https://github.com/kubernetes-client/python/issues/765
    
    The commit fixes this issue by setting TMPDIR to /var/run/dcmanager
    when sm starts dcmanager-manager.
    
    Change-Id: Ib147c2ab26e303032e18da51a506e3768bc471e0
    Closes-Bug: 1883599
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit dee5e270a3752ac47f4282cb12a5c273e093068b
Author: Bin Qian <bin.qian@windriver.com>
Date:   Tue Jun 16 16:51:55 2020 -0400

Ensure subcloud cert data is intact
    
    In reported cases subcloud bootstrap only received tls.key. With tls.crt
    and root_ca_crt are both empty.
    Solution is wait for the data values are not empty.
    
    Change-Id: I01d71f5d000247f03d2a32eb37761f3e186ac863
    Closes-Bug: 1883777
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit ed652edb38414343528bd78378fde06f7af5b66a
Author: Bin Qian <bin.qian@windriver.com>
Date:   Fri Jun 12 10:50:09 2020 -0400

Update dc-cert subcloud status endpoint when subcloud online
    
    Set it to in-sync and will be replaced with dc-cert validation
    result (coming soon).
    Ensure updating dc-cert status before subcloud is unmanaged.
    This is because the subcloud is bootstrap with in-sync dc-cert.
    And dc-cert is audit when subcloud is online (before set to
    managed).
    
    Change-Id: I2e6aa597e53da5e26f44430767eb6fd4d3e784ee
    Story: 2007347
    Task: 40070
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit c1da0762d58182171c12aca783c8f00c9ef2ac2b
Author: Sharath Kumar K <sharath.kumar@intel.com>
Date:   Tue May 5 06:28:22 2020 +0200

Tox and Zuul job for the bandit code scan in stx/distributedcloud
    
    Setting up the bandit tool for the scanning of HIGH severity issues
    in the python codes under Starlingx/distributedcloud folder.
    Expecting this merge will enable zuul job for CI/CD of bandit scan.
    
    Configuration files:
    1. tox.ini for adding bandit environment and command.
    2. test-requirements.txt for adding bandit version.
    3. .zuul.yaml file for adding bandit job and configuring under
       check job to run code scan every time before code commit.
    
    Test:
    Run tox -e bandit command inside the fault folder to validate the
    bandit scan and result.
    
    Please note:
    Changes will be implemented in batches and  this is Batch6 change.
    
    Story: 2007541
    Task: 39648
    Depends-On: https://review.opendev.org/#/c/721294/
    
    Change-Id: I6fdfeacdb000faafc73c44eef352bf2abd38a9a2
    Signed-off-by: Sharath Kumar K <sharath.kumar@intel.com>

commit cd12fc31a1702f08f77fedcaa6386825d9ae3c6c
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Jun 11 08:43:21 2020 -0500

Add reboot handling to unlock state for DC upgrades
    
    When an unlock is triggered, the host is rebooted.
    
    The maximum amount of time for this state to complete is 1 hour.
     - Reboot time (30 minutes)
     - pre and post reboot time (30 minutes)
    
    The unlock state handler for upgrade orchestration will
    now loop and ignore failures for a period of time, while
    waiting for the host to become unlocked / enabled.
    
    Change-Id: Ie7ad8dbfecd6ffbe241d098c8677a8e25f400b07
    Story: 2007403
    Task: 40048
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 4fd65e9913d28bd097b16ad3bfb80c04ca6b5419
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon May 25 13:51:18 2020 -0400

CLI command to deploy a subcloud
    
    If deployment failed, the user has no option than to delete and
    re-add it. If the user was re-adding the subcloud without
    re-installing, it would further result in a bootstrap failure.
    Thus, to simplify things, a new CLI command is provided to allow
    re-deployment. Furthermore, if the user still chooses to delete
    the subcloud and re-add it without a re-install, a better error
    message is provided asking them to re-install the host.
    
    CLI:
    dcmanager subcloud reconfig <id/name> --deploy-config <file>
    
    Test Cases:
    1) Successfully add a subcloud with or without deployment option
    2) Fail to re-add a subcloud without re-installation after a failed
       deployment
    3) Re-deploy with new CLI command after successful and unsuccessful deployment
    4) Re-deploy with new CLI command before and after the subcloud is unlocked
    5) Test new CLI command by passing wrong parameters
    
    Change-Id: I9fe7e3791e3887160668281048c3c12a7f40c2af
    Partial-Bug: 1864756
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 66f9341692336e86ee74712dc05e816fae5397ba
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 9 00:34:14 2020 -0400

Manage subcloud only when deploy_status is "complete"
    
    Subcloud can be managed only if the deploy status is "complete"
    i.e. either subcloud is successfully bootstrapped or subcloud is
    successfully deployed. This will prevent partially configured
    subclouds from being managed and put into service.
    
    Change-Id: Ic4562c250e651f6df64f368db641d7c58aee887c
    Closes-Bug: 1882786
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 5e5c416a9b56d6c7385e6883a73c8b29ebce4709
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Jun 2 15:51:32 2020 -0500

Implement import-load and starting-upgrade strategy states
    
    This commit allows the state machine to be defined through an
    ordered list of states. The states are mapped to by a dictionary
    to state handler classes.
    
    This greatly simplifies the software upgrade orch thread so that it
    can be refactored in the future to share functionality with other orch
    thread strategy types.
    
    The following states have been implemented:
     - 'importing load'
     - 'starting upgrade'
     - 'activating upgrade'
     - 'completing upgrade'
    
    The 'installing license' state now uses this design pattern.
    Its functionality and unit tests are now in their own files.
    
    The 'unlocking' state requires additional changes for the
    'activating' and 'completing' states to pass.
    
    Change-Id: Iedc40bdea406461dd4c2ce3887a37be6ec3c999f
    Story: 2007403
    Task: 40001
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 70f4dc8537ae89cf2eb70a78bda64073f13b02a1
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Jun 9 13:43:04 2020 -0400

Subcloud deploy fails
    
    Deploy config generated by dcmanager does not truncate the existing
    file after it has been overridden. This leaves some invalid content
    at the end of the newly generated file. Fixes have been made for this.
    
    Change-Id: I6c7e7fbacb1ee323c3675076083e046498698425
    Closes-Bug: 1882826
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit efcfc7a45a170c51ea5094fd62cb7047b24fc92d
Author: Andreas Jaeger <aj@suse.com>
Date:   Thu Jun 4 14:16:23 2020 +0200

Switch to newer openstackdocstheme and reno versions
    
    Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
    these versions will allow especially:
    * Linking from HTML to PDF document
    * Allow parallel building of documents
    * Fix some rendering problems
    
    Update Sphinx version as well.
    
    Disable openstackdocs_auto_name to use 'project' variable as name.
    
    Change pygments_style to 'native' since old theme version always used
    'native' and the theme now respects the setting and using 'sphinx' can
    lead to some strange rendering.
    
    openstackdocstheme renames some variables, so follow the renames
    before the next release removes them. A couple of variables are also
    not needed anymore, remove them.
    
    See also
    http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
    
    Change-Id: I7da2bdaafe86b9eb930202bef3742571daa90e17

commit 125e465aff6801940f25762d7be2406d01349d18
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon May 25 12:16:55 2020 -0400

Extend sysinv api proxy to support load operations
    
    In this commit, the dcorch-sysinv-api-proxy is extended
    to support load import and delete requests.
    
    Upon receiving a successful response from sysinv api for load
    import/delete request, the load is saved to/removed from dc-vault
    accordingly.
    
    Tests:
      - Successful load import
      - Successful load delete (no subclouds have the deleted load)
      - Successful load delete (one subcloud has the deleted load)
      - Failed load import (exceeding limit)
      - Failed load import (bad signature)
      - Failed load delete (does not exist)
    
    Unit tests will be added via a separate commit under task 39903
    of story 2007082.
    
    Story: 2007403
    Task: 39840
    Depends-On: https://review.opendev.org/#/c/730632
    Change-Id: If2769b0faf093523e7e9bc97b8cdc6a5513534aa
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 63c007461d782158e62aa082d74c44c452969883
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon May 25 12:55:20 2020 -0500

Implementing Lock and Unlock upgrade strategy states
    
    Implement the states for lock and unlock.
    
    Adding stub methods and classes for the full upgrade strategy
    subcloud orchestration state machine.
    
    Adding unit tests to validate all lock and unlock success
    and failure scenarios.
    
    Change-Id: I5b8d89bc733b867e8568043b0c7e4a79d1925890
    Story: 2007403
    Task: 39874
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 2f49219858ed3c3a872dcaa337ba601a68fce7e4
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue May 26 16:20:45 2020 -0400

Combine the patch audit with the subcloud audit
    
    This commit moves the patch audit to dcmanager-audit process.
    Patch audit does not run at the same frequency as subcloud
    audit, its normal interval is approx. 5 mins, but it can be
    forced to run on the next audit interval by calling
    trigger_patch_audit RPC.
    
    Change-Id: I4d4fdd0cebe9f3f65241107c92bbc4d26c13c8f3
    Story: 2007267
    Task: 39864
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 30f2147c398eb56fb67cb7b777560b3a18529ba5
Author: Tao Liu <tao.liu@windriver.com>
Date:   Mon May 25 13:40:16 2020 -0400

Close any temp files after upload a patch
    
    Add a cleanup routine in the patch-api-proxy to close
    any temporary files that opened by third-party libraries,
    during a patch upload.
    
    Verified that the temp files are close and /scratch
    usage is not increased after a few of large patch
    upload.
    Verified that the temp files are close and /scratch
    usage is not increased after a failed upload.
    
    Change-Id: I68453653fe09d8b1ad80289ab3ff33e476b3acee
    Closes-Bug: 1880587
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 566076a0a8808ee958f5edb4fb9333f426bf1490
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Wed May 20 02:30:03 2020 -0400

Fix slow subcloud manage due to sysinv auth
    
    Synchronize keystone services project and sysinv user ids between the
    system controller and subcloud during subcloud bootstrap.  This fixes
    issue where the subcloud would go offline on initial manage for 1 hour
    until keystone token expires.
    
    Change-Id: I2b36861df0858920eed308bd8e9a12b49b68a191
    Closes-Bug: 1877419
    Closes-Bug: 1877584
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit 6c09914c1140bd7d7e3988e7c42500c6d98df100
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 14 09:12:31 2020 -0500

Adding a license install step for subcloud upgrade
    
    The license install step queries the license on the
    system controller and installs it on the subclouds.
    
    If the system controller does not have a license,
    the license install state is skipped and the strategy
    proceeds to the next stage.
    
    On successful completion of the 'installing license' stage
    the strategy proceeds to the next stage which is the
    'importing load' stage.
    
    Change-Id: I61bc144bdf89302b4ba11fd35e82a9ff64d9c517
    Story: 2007403
    Task: 39780
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit bd4a76782d6b4896979c7c2fad467edc129b6ad7
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon May 18 17:40:18 2020 -0400

Use system software version for load audit.
    
    The current audit logic is based on the software version
    of the active load as opposed to the software version of
    the system. As a result, the subclouds will be out-of-sync
    shortly after the first controller has been upgraded.
    
    In this commit, the audit task checks the system software
    version which is only updated after both controllers have
    been upgraded. This will ensure no premature subcloud
    upgrade during upgrade orchestration.
    
    Tests:
      - Modify the system software version of the central
        cloud to simulate a software upgrade and verify that
        all subclouds are out-of-sync after the load audit.
      - Revert the above change and verify that all subclouds
        are back in-sync after the load audit.
      - Modify the system software version of one of the
        subclouds and verify that it is out-of-sync after
        the load audit.
    
    Closes-Bug: 1879396
    Change-Id: Ia10d561730f66f63f3e3113d8c3c56270051e7f3
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

tags:

added: in-f-centos8

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

auto-github-kubernetes-client-python #765
[closed lifecycle/rotten] Edit

Bug watches keep track of this bug in other bug trackers.