Reviewed:  https://review.opendev.org/716140
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=04b49dd093ab850f4520cdb85638221120dd7568
Submitter: Zuul
Branch:    f/centos8

commit 25c9d6ed3861f2d783404fcf84b186441ab9cd4d
Author: albailey <email address hidden>
Date:   Wed Mar 25 15:43:32 2020 -0500

    Removing ddt from unit tests
    
    This cleanup should assist in transitioning to
    stestr and fixtures, as well as py3 support.
    
    The ddt data is primarily unused, only subcloud, route
    and endpoints were being loaded.
    
    The information in the data files was out of date,
    and not necessarily matching the current product model.
    
    Story: 2004515
    Task: 39160
    Change-Id: Iddd7ed4664b0d59dbc58aae5c3fedd74c9a138c0
    Signed-off-by: albailey <email address hidden>

commit 7f3827f24d2fb3cb546d3caf71d505d23187b0dc
Author: Tao Liu <email address hidden>
Date:   Thu Mar 12 09:46:29 2020 -0400

    Keystone token and resource caching
    
    Add the following misc. changes to dcorch and dcmanager components:
    - Cache the master resource in dcorch audit
    - Consolidate the openstack drivers to common module, combine the
      dcmanager and dcorch sysinv client. (Note: the sdk driver that
      used by nova, neutron and cinder will be cleaned as part of
      story 2006588).
    - Update the common sdk driver:
      . in order to avoid creating new keystone client multiple times
      . to add a option for caching region clients, in addition to the
        keystone client
      . finally, to randomize the token early renewal duration
    - Change subcloud audit manager, patch audit manager,
      and sw update manager to:
      utilize the sdk driver which caches the keystone client and token
    
    Test cases:
    1. Manage/unmanage subclouds
    2. Platform resources sync and audit
    3. Verify the keystone token is cached until the token is
       expired
    4. Add/delete subclouds
    5. Managed subcloud goes offline/online (power off/on)
    6. Managed subcloud goes offline/online (delete/add a static route)
    7. Apply a patch to all subclouds via patch Orchestration
    
    Story: 2007267
    Task: 38865
    
    Change-Id: I75e0cf66a797a65faf75e7c64dafb07f54c2df06
    Signed-off-by: Tao Liu <email address hidden>

commit 3a1bf60caddfa2e807d4f5996ff94fea7dde5477
Author: Jessica Castelino <email address hidden>
Date:   Wed Mar 11 16:23:21 2020 -0400

    Cleanup subcloud details when subcloud add fails
    
    Failure during add subcloud prevents subcloud from being added again
    with the same name as the subcloud details are not cleaned up
    properly. Fixes have been added for proper cleanup of dcorch database
    tables, ansible subcloud inventory files, keystone endpoints, keystone
    region, and addn_hosts_dc file when failure is encountered.
    
    Test cases:
    1. Add subcloud
    2. Add subcloud with "--deploy-playbook"
    3. Delete subcloud
    4. Raise explicit exception in dcorch/objects/subcloud.py
    5. Raise explicit exception in dcmanager/manager/subcloud_manager.py
    
    Change-Id: Iedf172c3e9c3c4bdb9b9482dc5d46f072b3ccf61
    Closes-Bug: 1862774
    Signed-off-by: Jessica Castelino <email address hidden>

commit 1e588cbefa789bf8546da708470e271f01dcd593
Author: Andy Ning <email address hidden>
Date:   Wed Feb 26 16:04:33 2020 -0500

    Support multiple CA certificates synchronization in DC
    
    This update enhanced dcorch and sysinv API proxy to support multiple CA
    certificates synchronization in DC system. The support utilizes the
    updated sysinv certificate install API and the new certificate
    uninstall API.
    
    Closes-Bug: 1861438
    Closes-Bug: 1860995
    Depends-On: https://review.opendev.org/#/c/711538/
    Change-Id: I407314b913ae5a56bb714b39484aea3263a41d19
    Signed-off-by: Andy Ning <email address hidden>

commit dbd92e06500567ce7086fa172cdc745e1c62c6bb
Author: Jessica Castelino <email address hidden>
Date:   Tue Mar 10 15:04:59 2020 -0400

    Increase ThreadGroup size for patch orchestration
    
    The default ThreadGroup size is 10. In order to allow for more
    parallelism when doing patch orchestration across many subclouds,
    this is being increased to 100.
    
    Change-Id: I82ace0d616d1455fd65e1583c694a71f8dede721
    Story: 2007267
    Task: 38913
    Signed-off-by: Jessica Castelino <email address hidden>

commit 48268f420e1bbfbdf284fc3ef513626d1d2b051c
Author: Jessica Castelino <email address hidden>
Date:   Wed Mar 4 16:55:04 2020 -0500

    Remove remote logging configuration
    
    Remote logging is removed from dcorch logs to avoid
    sync of unnecessary system data
    
    Story: 2007267
    Task: 38970
    Change-Id: I36fade7ff4a87855207f570f103b7e1b8fc1262a
    Partial-Bug: 1857069
    Signed-off-by: Jessica Castelino <email address hidden>

commit 4fb11314833cf1b83ef18c332e151b129ea73aaf
Author: Tao Liu <email address hidden>
Date:   Mon Mar 9 18:12:18 2020 -0400

    Support OAM on vlan and pulling rvmc from local registry
    
    Add 'vlan=' option to anaconda boot options, and ensure 'ip=' option
    using a vlan interface when the bootstrap_vlan parameter is provided.
    
    The rvmc image, which had been pulled to the local registry during
    bootstrapping, was pulled from docker.io when the rvmc job/pod
    was launched. This update adds the local registry prefix to the
    image name override.
    
    Note: The OAM network on vlan configuration has been tested by the
    customer.
    
    Closes-Bug: 1866670
    Closes-Bug: 1866669
    
    Change-Id: I9ad27742cf4c380749cbad68ecaee165d114b11c
    Signed-off-by: Tao Liu <email address hidden>

commit e39a56cdc6e5fd63461d7e9ae25ebe617ccab94e
Author: Jessica Castelino <email address hidden>
Date:   Mon Mar 9 17:01:30 2020 -0400

    Fixed a typo in StarlingX API reference document
    
    Change-Id: Ie317ab71f356bf8c74984e45784c39daea53e7e2
    Signed-off-by: Jessica Castelino <email address hidden>

commit 3c8d6bc7f56c97350ca255502462d565eaed38b2
Author: Jessica Castelino <email address hidden>
Date:   Fri Mar 6 17:09:27 2020 -0500

    Reduce dcorch log size
    
    Removes unnecessary dcorch info logs to reduce log size
    
    Change-Id: Ib8b3b31c4c174b85cf95cb1a4fc06b3ae10f4d32
    Story: 2007267
    Task: 38740
    Depends-On: https://review.opendev.org/#/c/711775/
    Closes-Bug: 1857069
    Signed-off-by: Jessica Castelino <email address hidden>

commit fe3ef48558f4ba58981572a73b876da06c89220d
Author: Jessica Castelino <email address hidden>
Date:   Mon Mar 2 16:20:27 2020 -0500

    Show OAM Floating IP for Subcloud
    
    Extends the dcmanager subcloud show command to display the floating
    OAM IP of the subcloud from the system controller.
    
    Change-Id: Ib421ea65660ed77a241798831c83cbaf1710f9e8
    Story: 2007267
    Task: 38896
    Signed-off-by: Jessica Castelino <email address hidden>

commit 501fa35af2780d57329478fe0ef0a6037ef8bf73
Author: Al Bailey <email address hidden>
Date:   Tue Jan 21 17:04:56 2020 -0600

    Update pylint for distributedcloud
    
    The pylint that was specified as an upper constraint was not
    as strict as the newer versions.
    
    - Fixed or suppressed any new error codes being reported.
    Suppressed error codes should be fixed by additional reviews.
    
    - added a zuul target for pylint
    
    - Added dcdbsync to the folders being processed by pylint.
    
    - Removed the env:UPPER_CONSTRAINTS_FILE from tox.ini so that zuul
    does not override the upper constraints with an newer version.
    
    - Needed to add greenlet to the extension-pkg-whitelist since
    runtime introspection of that component is not possible.
    
    - oslo-messaging get_transport deprecated 'aliases' in pike, and
    removed it in rocky. The packaged version in STX is 5.30.6
    which still supports aliases. This requires us to explicitly
    run tox with the version that we ship (pre-stein). This is
    controlled by a local upper-constraints.txt file.
    Related-Bug: 1865054
    
    - Paste file handling reports a pylint issue which has been suppressed
    Related-Bug: 1865085
    
    Story: 2004515
    Task: 38882
    Change-Id: Ie7c8c2adab1eeb9100159a2aa29968a0b557e2e4
    Signed-off-by: Al Bailey <email address hidden>

commit abe9efce41d3b94c12aeaff14f013c5c25b76d42
Author: Tao Liu <email address hidden>
Date:   Sun Feb 9 20:47:16 2020 -0500

    Keystone token and client caching
    
    This update is to change the platform SDK driver to cache the keystone
    token and client sessions, also to add the fm client to the platform
    SDK in order to remove the sdk.py once nova, neutron and cinder
    references are removed (as part of story 2006588).
    
    The keystone client/token is cached per-region vs. the client session,
    which is cached per-region & per-thread. This is because, simultaneous
    access from different greenthreads to the client socket is not allowed.
    
    The initial sync does not use the cached clients as requesting a
    new token is required for each region.
    
    In addition, this update modifies the EndpointCache class to load
    the auth plugin once.
    
    Test cases:
    1. Manage/unmanage subclouds
    2. Platform resources and alarm summary sync
    3. Platform resources and alarm summary audit
    4. Verify the keystone client/token is cached until the token is
       expired
    5. Add/delete subclouds
    6. Managed subcloud goes offline/online (power off/on)
    7. Managed subcloud goes offline/online (delete/add a static route)
    
    Story: 2007267
    Task: 38709
    
    Change-Id: I0842a79838ea0f7a6c16f3b1e69ad0eb1357018a
    Signed-off-by: Tao Liu <email address hidden>

commit bde6982ded5ae8995d6ba0b758ea990086b7fa05
Author: Al Bailey <email address hidden>
Date:   Tue Jan 21 16:31:33 2020 -0600

    Updating flake8 version for distributed cloud
    
    This also fixes the linters target which fails when yamllint
    does not find any inputs.
    
    Most new failures are being suppressed for now, and will be
    cleaned up by future submissions.
    
    Story: 2004515
    Task: 38739
    Change-Id: I8bddba83408be19a487837e326c86432579c50db
    Signed-off-by: Al Bailey <email address hidden>

commit 9180e7df84840745138827688cc8a482dd02f09c
Author: Robert Church <email address hidden>
Date:   Thu Feb 6 16:44:48 2020 -0500

    Add various locking support to DCManager
    
    Support the following lock updates in DCManager:
     - Provide a function decorator in common/utility.py for a synchronized
       lock that supports both external locks and internal fair locks. This
       decorator is setup, by default, for external locks.
     - Refactor update_subcloud_endpoint_status() so that a common private
       method is provided that is suitable for locking.
     - Update subcloud_manager.py to provide a function decorator to produce
       an internal fair lock based on a unique subcloud name. This decorator
       is specifically designed to be used with
       _update_subcloud_endpoint_status(). This will ensure that the
       multi-threaded DCManager process will only update subcloud endpoint
       information in a synchronized manner.
     - Provide an API lock to the SubcloudsController for the post, patch,
       and delete operations
    
    Update distributedcloud requirements and spec file to require
    oslo.concurrency >= 3.29.1. This is the latest version supported by the
    Openstack Stein and is a version containing fair lock support.
    
    Update unit tests:
     - Added unit test for update_subcloud_endpoint_status. This verifies
       high level functionality and the calling of fair locks based on the
       unique subcloud name.
     - Fixed intermittent failure seen when executing the add_subcloud unit
       test by mocking thread.Threading.
     - Leverage the use of oslo_concurrency's behavior to use the
       OSLO_LOCK_PATH environment variable if the lock_path config option is
       not set. Currently this is not set as we specify a hard coded
       external lock path at runtime. This allows us to set the lock path
       for tox tests via the test environment.
    
    Change-Id: Id1902e8553408cbdd60b648efc39d59e8edcdb55
    Depends-On: https://review.opendev.org/#/c/707188/
    Closes-Bug: #1855359
    Signed-off-by: Robert Church <email address hidden>

commit 0389c7fbb1630988acd385140c9fc16835aae090
Author: Bart Wensley <email address hidden>
Date:   Tue Feb 11 15:21:09 2020 -0600

    Fix subcloud manage/unmanage issues caused by identity sync
    
    Recently identity (keystone) sync functionality was added to the
    dcorch. This changed the behaviour of the update_subcloud_states
    RPC. The dcmanager expects this RPC to be handled quickly and
    a reply sent almost immediately (timeout is 60s). Instead, the
    dcorch is now performing an identity sync when handling this
    RPC, which involves sending multiple messages to a subcloud and
    waiting for replies. This causes the update_subcloud_states RPC
    to time out sometimes (especially if a subcloud is unreachable)
    and the dcmanager/dcorch states to get out of sync, with no
    recovery mechanism in place.
    
    To fix this, I have create a new initial sync manager in the
    dcorch. When the dcorch handles the update_subcloud_states RPC,
    it will now just update the subcloud to indicate that an initial
    sync is required and then reply to the RPC immediately. The
    initial sync manager will perform the initial sync in the
    background (separate greenthreads) and enable the subcloud when
    it has completed. I also enhanced the dcmanager subcloud audit
    to periodically send a state update for each subcloud to the
    dcorch, which will correct any state mismatches that might
    occur.
    
    Change-Id: I70b98d432c3ed56b9532117f69f02d4a0cff5742
    Closes-Bug: 1860999
    Closes-Bug: 1861157
    Signed-off-by: Bart Wensley <email address hidden>

commit b3c2086acf9295b5b11bac4cdb3c918423cadc14
Author: Tao Liu <email address hidden>
Date:   Wed Feb 12 10:07:36 2020 -0500

    Generate a hybrid bootloader from ISO
    
    Update subcloud installer to use the gen-bootloader-iso.sh
    utility to generate a hybrid bootloader from ISO
    
    Add no_check_certificate option to allow the user to disable
    the certificate check.
    
    Depends-On: https://review.opendev.org/#/c/707049/
    Story: 2006980
    Task: 38465
    
    Change-Id: I83b5d09ccd4076ee44360446b1ae1706c06cebdf
    Signed-off-by: Tao Liu <email address hidden>

commit f3f6872b50822211a78888b4015ff9370be752b3
Author: Tao Liu <email address hidden>
Date:   Mon Feb 3 15:53:14 2020 -0500

    Distributed Cloud: Large patch upload failed
    
    The third-party modules that used by the proxy service utilize
    the tempfile module to create interim files, under the default
    temp location.  As a result, a large patch upload failed due to
    "No space left on device".
    
    This update sets TMPDIR environment variable to
    /scratch/patch-api-proxy-tmpdir in the patch proxy service,
    so that the temp files will not use the default location.
    
    Change-Id: Iab4a8fefbab74047b3ec9f318397b21f75c1c5cb
    Closes-Bug: 1861329
    Signed-off-by: Tao Liu <email address hidden>

commit 6b95eb951a26fa05142143fa51d3cde99738dbee
Author: Bin Qian <email address hidden>
Date:   Wed Feb 5 14:41:17 2020 -0500

    Adding job to upload commits to GitHub
    
    Add job to publish distcloud repo to GitHub
    
    Change-Id: I1825e823a5be37ede21b658297a108c0328b6fbf
    Story: 2007252
    Task: 38667
    Signed-off-by: Bin Qian <email address hidden>