StarlingX

Distributed Cloud: Many clients received http 500 errors in batch deployment

Bug #1865573 reported by Tee Ngo on 2020-03-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Jessica Castelino

Bug Description

Brief Description
-----------------
The add subcloud REST api returns http 500 errors

Severity
--------
Major - the add subcloud requests are still being processed by the backend but these errors will give the users the impression that their requests have failed. This also breaks automated batch subcloud deployment.

Steps to Reproduce
------------------
Send 50 add subcloud requests via CLI or REST API simultaneously

Expected Behavior
------------------
REST API request - http 200 resonse
CLI request - a confirmed response with subcloud UUID

Actual Behavior
----------------
REST API - http 500 response
CLI - ERROR Unable to add subcloud

A sample timed out log in dcmanager.log

Traceback (most recent call last):
   File ""/usr/lib/python2.7/site-packages/dcmanager/api/controllers/v1/subclouds.py"", line 447, in post
     return self.rpc_client.add_subcloud(context, payload)
   File ""/usr/lib/python2.7/site-packages/dcmanager/rpc/client.py"", line 68, in add_subcloud
     payload=payload))
   File ""/usr/lib/python2.7/site-packages/dcmanager/rpc/client.py"", line 56, in call
     return client.call(ctxt, method, **kwargs)
   File ""/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py"", line 465, in call
     return self.prepare().call(ctxt, method, **kwargs)
   File ""/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py"", line 169, in call
     retry=self.retry)
   File ""/usr/lib/python2.7/site-packages/oslo_messaging/transport.py"", line 123, in _send
     timeout=timeout, retry=retry)
   File ""/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py"", line 566, in send
     retry=retry)
   File ""/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py"", line 555, in _send
     result = self._waiter.wait(msg_id, timeout)
   File ""/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py"", line 447, in wait
     message = self.waiters.get(msg_id, timeout=timeout)
   File ""/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py"", line 335, in get
     'to message ID %s' % msg_id)
MessagingTimeout: Timed out waiting for a reply to message ID c654c39f48624cffbb56e3a866b92bea

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
IPv6 distributed cloud

Branch/Pull Time/Commit
-----------------------
Feb. 22 master

Last Pass
---------
Not certain when this test case was verified

Timestamp/Logs
--------------
dcmanager logs attached

Test Activity
-------------
Evaluation

Tags:

Revision history for this message

Tee Ngo (teewrs) wrote on 2020-03-02:

batch_add_subclouds.tgz Edit (4.5 MiB, application/x-tar)

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-03-04:

stx.4.0 / medium priority - issue appears to be tied to the # of subclouds being added simultaneously. Should be investigated/addressed.

tags:	added: stx.4.0 stx.distcloud
Changed in starlingx:
importance:	Undecided → Medium
status:	New → Triaged
assignee:	nobody → Tao Liu (tliu88)

Revision history for this message

Bart Wensley (bartwensley) wrote on 2020-03-26:

Adding some more info based on investigation and discussions...

The handling of the add_subcloud by the dcmanager needs to be asynchronous in order to support parallel add requests. This will require a bit of restructuring (the addition of the subcloud to the dcmanager DB would have to happen in the API) and we may need to look at adding a new state for this “adding” phase.

Bart Wensley (bartwensley) on 2020-04-28

Changed in starlingx:
assignee:	Tao Liu (tliu88) → Jessica Castelino (jcasteli)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-01: Fix proposed to distcloud (master)

Fix proposed to branch: master
Review: https://review.opendev.org/724926

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-12: Fix merged to distcloud (master)

Reviewed: https://review.opendev.org/724926
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=ae73f0dd585c53c0242d6da7f7ef717e24ec9ddd
Submitter: Zuul
Branch: master

commit ae73f0dd585c53c0242d6da7f7ef717e24ec9ddd
Author: Jessica Castelino <email address hidden>
Date: Thu Apr 30 10:15:35 2020 -0400

Make the add_subcloud RPC call asynchronous

    The add subcloud REST API returns HTTP 500 errors although the
    requests are being processed in the backend. This is because the
    work done while handling the add_subcloud RPC can take too long
    and cause the RPC to time out, which causes the POST to fail,
    even though dcmanager-manager continues to add the subcloud. Thus,
    a fix is added to make the add_subcloud RPC call asynchronous.

    Change-Id: I89d9ce8367ef124c77869dff309f6bb3a621df34
    Closes-Bug: 1865573
    Signed-off-by: Jessica Castelino <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-21: Fix proposed to distcloud (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/729815

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-03: Fix merged to distcloud (f/centos8)

Download full text (19.6 KiB)

Reviewed: https://review.opendev.org/729815
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=a39415ff84d71a54580eb8cbe885647e69279306
Submitter: Zuul
Branch: f/centos8

commit 5c8377047ba679ce88a0360315df494a74208bbf
Author: Tao Liu <email address hidden>
Date: Tue May 5 08:59:59 2020 -0500

Move subcloud audit to separate process

    Remove subcloud audit from dcmanager-manager process.
    Create dcmanager-audit process & associated files.
    Add new RPC calls for dcmanager-audit to notify dcmanager
    subcloud availability and sync endpoint type changes.
    Update dcmanager to handle availability and sync endpoint
    type updates from dcmanager-audit.
    Subcloud audit interval will be reduced to 20 seconds.
    Create/update unit tests, to verify the implementation
    changes.

Story: 2007267
Task: 39637

Change-Id: Iff408166753f22ce3616d34e267ca1155ac43042
Signed-off-by: Tao Liu <email address hidden>

commit d46516c46d24f8fea6ea71fdbe2f3fa2d296eb4d
Author: albailey <email address hidden>
Date: Wed May 13 14:00:12 2020 -0500

Enable python3 unit tests as part of zuul

    The existing py27 unit tests were not all passing in py36,
    however now they are and so the zuul check and gate for py36
    have been added.

    Change-Id: Ie293ec69a04e6fd657f960aa9a135c428138b4b4
    Story: 2004515
    Task: 39768
    Signed-off-by: albailey <email address hidden>

commit dbf603b4c45e865e56a21d3b14d94bcc8d5f455c
Author: albailey <email address hidden>
Date: Mon May 11 10:29:45 2020 -0500

Reduce the number of suppressed pylint warnings

    All pylint warnings were being suppressed by a wildcard.
    This commit only suppresses the warnings that are failing and
    prevents checks that would pass from being broken in later commits.

The warnings being suppressed can be resolved individually
by later submissions based on priority where appropriate.

This commit also specifies python3 for pylint which has
more recent checks.

    Change-Id: Ie29aeb0ea3e9dcb671af67f38e9a3f919ea7111e
    Story: 2004515
    Task: 39734
    Signed-off-by: albailey <email address hidden>

commit 15fb58f45c0f552eccd9c27ba023dbea560f27f2
Author: albailey <email address hidden>
Date: Fri May 8 13:20:23 2020 -0500

Enhance Upgrade strategy to use endpoint audit status

    The distributed cloud audit was updated to include 'load'
    endpoint status, so the upgrade strategy is now able to
    make use of that information when constructing a strategy.

    Change-Id: I69eb4d98b9abf38b329e13fb116fc098db2bd736
    Story: 2007403
    Task: 39736
    Signed-off-by: albailey <email address hidden>

commit acc710093bb1c9581670d20b62c68ad669b3a3ad
Author: MCamp859 <email address hidden>
Date: Mon May 11 14:28:28 2020 -0400

Minor edits to test docs promote issue

Change-Id: I08ffc5e57b5b04c59102a6491f2fcdc256f16e0f
Signed-off-by: MCamp859 <email address hidden>

commit 6d4fa855462cc6faf2e962f9d825b832f2885aa3
Author: Tee Ngo <email address hidden>
Date: Mon May 4 23:53:20 2020 -0400

Extend subcloud audit...

Reviewed:  https://review.opendev.org/729815
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=a39415ff84d71a54580eb8cbe885647e69279306
Submitter: Zuul
Branch:    f/centos8

commit 5c8377047ba679ce88a0360315df494a74208bbf
Author: Tao Liu <tao.liu@windriver.com>
Date:   Tue May 5 08:59:59 2020 -0500

Move subcloud audit to separate process
    
    Remove subcloud audit from dcmanager-manager process.
    Create dcmanager-audit process & associated files.
    Add new RPC calls for dcmanager-audit to notify dcmanager
    subcloud availability and sync endpoint type changes.
    Update dcmanager to handle availability and sync endpoint
    type updates from dcmanager-audit.
    Subcloud audit interval will be reduced to 20 seconds.
    Create/update unit tests, to verify the implementation
    changes.
    
    Story: 2007267
    Task: 39637
    
    Change-Id: Iff408166753f22ce3616d34e267ca1155ac43042
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit d46516c46d24f8fea6ea71fdbe2f3fa2d296eb4d
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed May 13 14:00:12 2020 -0500

Enable python3 unit tests as part of zuul
    
    The existing py27 unit tests were not all passing in py36,
    however now they are and so the zuul check and gate for py36
    have been added.
    
    Change-Id: Ie293ec69a04e6fd657f960aa9a135c428138b4b4
    Story: 2004515
    Task: 39768
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit dbf603b4c45e865e56a21d3b14d94bcc8d5f455c
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon May 11 10:29:45 2020 -0500

Reduce the number of suppressed pylint warnings
    
    All pylint warnings were being suppressed by a wildcard.
    This commit only suppresses the warnings that are failing and
    prevents checks that would pass from being broken in later commits.
    
    The warnings being suppressed can be resolved individually
    by later submissions based on priority where appropriate.
    
    This commit also specifies python3 for pylint which has
    more recent checks.
    
    Change-Id: Ie29aeb0ea3e9dcb671af67f38e9a3f919ea7111e
    Story: 2004515
    Task: 39734
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit 15fb58f45c0f552eccd9c27ba023dbea560f27f2
Author: albailey <Al.Bailey@windriver.com>
Date:   Fri May 8 13:20:23 2020 -0500

Enhance Upgrade strategy to use endpoint audit status
    
    The distributed cloud audit was updated to include 'load'
    endpoint status, so the upgrade strategy is now able to
    make use of that information when constructing a strategy.
    
    Change-Id: I69eb4d98b9abf38b329e13fb116fc098db2bd736
    Story: 2007403
    Task: 39736
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit acc710093bb1c9581670d20b62c68ad669b3a3ad
Author: MCamp859 <maryx.camp@intel.com>
Date:   Mon May 11 14:28:28 2020 -0400

Minor edits to test docs promote issue
    
    Change-Id: I08ffc5e57b5b04c59102a6491f2fcdc256f16e0f
    Signed-off-by: MCamp859 <maryx.camp@intel.com>

commit 6d4fa855462cc6faf2e962f9d825b832f2885aa3
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon May 4 23:53:20 2020 -0400

Extend subcloud audit to include loads audit
    
    In this commit, the existing patching audit logic is extended
    to include subcloud loads audit every other cycle since it already
    has to retrieve subcloud loads info in order to determine patching
    sync status.
    
    Tests:
     - Add and manage a few subclouds.
     - Unmanage and remove a couple of subclouds.
     - Verify that loads audit only occurs every other cycle.
     - Simulate update in progress in one of the subclouds and verify
       its sync status.
     - Simulate old software version in one of the subclouds and
       verify its sync status.
    
    Story: 2007403
    Task: 39646
    Change-Id: I496086bf90070a732b10bb7597c1af1f1fcd7952
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit ff0c90231459965faa5fcc3a09a3dbb0a3ee6c04
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Apr 22 06:34:21 2020 -0500

Upgrade strategy support
    
    This feature is based on sw-update-strategy and attempts
    to reuse and refactor that code where appropriate so that
    more than just patch can be applied using a strategy.
    
    The 'upgrade' strategy type has a new orchestration thread class.
    
    The strategy 'lock' object is shared among all orchestration threads.
    
    Change-Id: Ic1ec7b3665159295005db2133b136f15c62bbddb
    Story: 2007403
    Task: 39655
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit ae73f0dd585c53c0242d6da7f7ef717e24ec9ddd
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu Apr 30 10:15:35 2020 -0400

Make the add_subcloud RPC call asynchronous
    
    The add subcloud REST API returns HTTP 500 errors although the
    requests are being processed in the backend. This is because the
    work done while handling the add_subcloud RPC can take too long
    and cause the RPC to time out, which causes the POST to fail,
    even though dcmanager-manager continues to add the subcloud. Thus,
    a fix is added to make the add_subcloud RPC call asynchronous.
    
    Change-Id: I89d9ce8367ef124c77869dff309f6bb3a621df34
    Closes-Bug: 1865573
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 0bbf612a09b63c54cc6fa0ea4c74f1e1c8ea4afd
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Apr 21 11:23:26 2020 -0400

Avoid synching SSL certificates to subclouds
    
    The SSL certificate on the SystemController will have a CN and SAN
    with OAM IP Address and any related DNS Names for OAM IP Address. If
    put on the subclouds, even if the certificate was signed by a trusted
    CA, when any client connects to the subcloud OAM IP, the certificate
    that comes back will not pass certificate validation because the
    certificate does not apply to the subcloud's IP Address. For this
    reason, a fix is put in place to stop synching SSL certificates
    to the subcloud.
    
    Change-Id: I2f5625eeeb30b9fca025b0d0b4a15d926b700d4d
    Closes-Bug: 1865643
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 7f156476f9fb7f589d1c6f4ad45f60668ef9015b
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Wed Apr 15 15:05:43 2020 -0400

Prevent unlimited growth of dcorch database
    
    In this commit:
      - completed orch request is soft deleted right away.
      - failed orch requests that have not been soft deleted
        are soft deleted at the end of the next audit cycle.
      - prior to a sync run, if the managed subcloud has any
        queued, in-progress, and/or failed requests, its
        sync status will be set to out-of-sync.
      - a cron job that runs 20 minutes in the first hour
        daily (another commit in stx-puppet repo) will purge
        all soft deleted orch requests that are 3 days old,
        their orch jobs and resources.
    
    Tests:
      - Identity and platform config changes with and
        without proxy.
      - Database inspection after config changes and after
        soak.
      - Purge of large number of orch requests, orch jobs
        and resources.
    
    Story: 2007267
    Task: 39044
    Change-Id: If5c75b9af3e1e20c010308919cb6be8f11575546
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit e75e33d97a2df04a2acca1587845b578f5fbc18f
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Fri Apr 24 11:22:42 2020 -0400

Fix bouncing identity sync status on first manage
    
    When a subcloud has been managed for the first time, it's been observed
    that its sync status bounces from in-sync to out-of-sync to in-sync.
    This is due to identity sync status going out of sync due to identity
    roles and admin user requiring two rounds of syncing to fully match
    details between system controller and subcloud.  To fix, force roles and
    admin user to sync during subcloud initial sync phase.
    
    Change-Id: Ica0c6d8684055e1f1c1c5719b9b0991a75b75bad
    Closes-Bug: 1870434
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit 7122dd597ff8c6400c999e225ca5cf2925f1563c
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Apr 29 18:54:37 2020 -0400

Remove egg-info from additional packages
    
    The python egg-info files are currently being packaged in each of the
    DC RPMs, when they should only be in one. This update drops them from
    the dcmanager, dcorch, and dcdbsync RPMs.
    
    Change-Id: Id526f7b6cf2221c7adb795929200262601139473
    Partial-Bug: 1875978
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit ea18400f4ee4b6afb870fa6db12f01077a98a7be
Author: Bin Qian <bin.qian@windriver.com>
Date:   Thu Apr 16 14:36:58 2020 -0400

Change subcloud admin endpoint to https
    
    Change subcloud admin endpoint to https, as the subcloud
    admin endpoints are changed to https in the dependent
    change lists.
    
    Depends-on: https://review.opendev.org/#/c/720224/
    Depends-on: https://review.opendev.org/#/c/720852/
    
    Story: 2007347
    Task: 39449
    
    Change-Id: Ie021f3476493b5cc5e397e0a9bff8ae4a6ee9079
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit 96072689a390513d7a2bc1ddcc5918af58c536b5
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Apr 24 16:37:50 2020 -0400

Rename the existing /opt/patch-vault filesystem to /opt/dc-vault
    
    The filesystem /opt/patch-vault is renamed to /opt/dc-vault so that
    it can be re-used to store FPGA images and software loads. Thus,
    paths have been changed, however, variables haven't been renamed.
    
    Change-Id: Ib229336d260fdc70cbf0672167da571723501515
    Story: 2006740
    Task: 39550
    Depends-On: https://review.opendev.org/#/c/723007/
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 1b7c111a48e7794e7b2a9fdd54a6a46f69c78bc2
Author: Bin Qian <bin.qian@windriver.com>
Date:   Tue Apr 7 23:23:58 2020 -0400

Pass root cert and intermediate cert to subcloud
    
    When a subcloud is created, create an intermediate CA certificate and
    pass the certificate and key pair to subcloud for subcloud to create
    its intermediate CA.
    Also pass root CA certificate to subcloud so all certificates issued
    directly or indirectly by DC root CA are trusted.
    On the subcloud delete, delete the intermediate CA and certificate.
    
    Test cases:
    
    Add subclouds
      - intermediate cert and CA cert are passed to subcloud bootstrap
    Delete subclouds
      - intermediate cert and secret are deleted from cert-manager
    
    Story: 2007347
    Task: 39432
    
    Depends-on: https://review.opendev.org/#/c/720270
    
    Change-Id: I899ddea6b6d2f4fae7a5209251d00bca315a2aa4
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit c890f8314ecc88d89cb0a4ae5f0e423580d94d6b
Author: Tao Liu <tao.liu@windriver.com>
Date:   Thu Apr 23 11:49:02 2020 -0400

Display an error message when the deploy file is missing
    
    Subcloud add command fails without an error message if the
    deploy files were not uploaded to System Controller.
    
    This update ensures that an error message is displayed to
    inform users the pertinent files are missing.
    
    Closes-Bug: 1864508
    
    Change-Id: Ibfceb6bc9155dc60d8e8379966b71bb2c7f076f0
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 06e8c7d414ee9624ec948006ee19b7db2bcc75f1
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Apr 8 10:29:13 2020 -0400

Change central cloud to access subcloud via admin endpoints
    
    system controller is changed to access subcloud via admin endpoints,
    instead of internal endpoints.
    With upcoming changes that enable https on admin endpoints,
    to provide secured access.
    
    Story: 2007347
    Task: 39448
    
    Change-Id: Iaa0454acc0b3a0da4da0d51f44dc181808ea6035
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

commit 58a7186beae7b10866aecd53fcde5be34321a52c
Author: Tao Liu <tao.liu@windriver.com>
Date:   Thu Apr 16 10:37:32 2020 -0400

Support subcloud deploy upload the common files
    
    Add new REST APIs to upload and display the deploy manager
    common files on the System Controller.
    
    The deploy manager common files which include playbook,
    overrides and helm charts are uploaded to
    /opt/platform/deploy/<version>:
    /opt/platform/deploy/<version>/deploy_playbook_<original name>
    /opt/platform/deploy/<version>/deploy_overrides_<original name>
    /opt/platform/deploy/<version>/deploy_chart_<original name>
    
    Modify the subcloud post request to accept the bootstrap-values,
    install-values and deploy-config as file contents. The deploy
    config file is only used by the deploy manager and it is
    uploaded to /opt/dc/ansible.
    
    The information that used to create the overrides for the
    playbook are extracted and sent to the dcmanager, which include
    bootstrap values, install values and the full path of deploy
    file names, if the deploy-config is presented in the request.
    
    Testcases:
    REST APIs:
    1. curl -X POST -H "X-Auth-Token: $TOKEN" $APIURL/subcloud-deploy \
    -F deploy_playbook=@<full path of the playbook name> \
    -F deploy_overrides=@<full path of the override file name> \
    -F deploy_chart=@full path of the helm chart name>
    
    2. curl -X GET -H "X-Auth-Token: $TOKEN" $APIURL/subcloud-deploy
    
    3. curl -X POST -H "X-Auth-Token: $TOKEN" \
    $APIURL/subclouds \
    -F bootstrap_values=@<full path of the bootstrap override file> \
    -F sysadmin_password=<encoded password> \
    -F bootstrap-address=<bootstrap IP>
    
    4. curl -X POST -H "X-Auth-Token: $TOKEN" \
    $APIURL/subclouds \
    -F bootstrap_values=@<full path of the bootstrap override file> \
    -F install_values=@<full path of the install value file> \
    -F deploy_config=@<full path of the deploy config file> \
    -F sysadmin_password=<encoded password> \
    -F bmc_password=<encoded password> \
    -F bootstrap-address=<bootstrap IP> \
    
    CLI:
    1. dcmanager subcloud-deploy upload \
    --deploy-playbook <full path of the playbook name> \
    --deploy-chart <full path of the override file name> \
    --deploy-overrides <full path of the override file name>
    
    2. dcmanager subcloud-deploy show
    
    3. dcmanager subcloud add --bootstrap-address <IP>  \
    --bootstrap-values <full path of the bootstrap override> \
    --deploy-config <full path of the deploy config file> \
    
    4. dcmanager subcloud add --bootstrap-address <IP> \
    --bootstrap-values <full path of the bootstrap override> \
    --install-values <full path of the install value file> \
    
    5.dcmanager subcloud add --bootstrap-address <IP> \
    --bootstrap-values <full path of the bootstrap override> \
    --install-values <full path of the install value file> \
    --deploy-config <full path of the deploy config file> \
    
    Host swact and deploy of a subcloud
    
    Closes-Bug: 1864508
    
    Change-Id: I3ce0da6efb8c2d78a213647789fc6bdb3b348b2d
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit 30b9714844ee1c720796484872edacc95d71f5fc
Author: Tee Ngo <tee.ngo@windriver.com>
Date:   Mon Apr 13 11:34:09 2020 -0400

Eliminate redundant endpoints in central cloud
    
    This commit reduces the redundant endpoints in the central
    cloud. As a result, load to keystone service as part of
    subcloud add as well as endpoint table size is cut by 40%.
    
    Tests:
      - Successfully navigate to each panel and pane of the
        system controller/central cloud.
      - Successfully switch to a subcloud.
      - Successfully navigate to each panel and pane of a
        subcloud.
    
    Closes-Bug: 1869790
    Depends-On: https://review.opendev.org/#/c/719577
    Change-Id: I2c6fb389ef0550cf2d4235d6f4b4cf3cd9711143
    Signed-off-by: Tee Ngo <tee.ngo@windriver.com>

commit 5e884374d2e9d7a26a5b03c230fd59bca428af01
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Mar 3 13:55:18 2020 -0600

Adding DistributedCloud subcloud group feature
    
    A new subcloud group feature has been added as a way to
    logically group several subclouds.  This would allow a group
    of subclouds to be updated in parallel or in serial, or to
    limit how many are updated in parallel.
    
    A 'Default' subcloud group is created in all installations,
    and all subclouds are automatically members of that
    subcloud group unless explicitly updated to reference another.
    
    The Default subcloud group cannot have its name changed, and
    other groups cannot use the same name.
    
    A subcloud group that has subclouds in it, cannot be deleted.
    
    Change-Id: Id0f232966c0128ecf6d87f941abcd42ff5a1a5c1
    Story: 2007518
    Task: 39301
    Signed-off-by: albailey <Al.Bailey@windriver.com>

commit b0636c1ed89d062bf777b22ad05512d655c13401
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Fri Mar 27 15:20:12 2020 -0400

Prevent sync states from bouncing when a subcloud is managed
    
    When a subcloud is first set to managed, the dcorch endpoint
    sync status keeps toggling between the two states - "in-sync"
    and "out-of-sync", before settling on "in-sync". This is
    because during the audit run, different endpoints (sysinv and
    identity) are done in separate greenthreads and each
    greenthread processes the resources (e.g. users, projects)
    in series. Thus, an enhancement is made to reduce the
    unnecessary messaging between dcorch and dcmanager, and to
    avoid the bouncing sync states.
    
    Change-Id: I98cec42eb5880d95d54182339dab6b5c951d6cfd
    Story: 2007267
    Task: 38996
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 085a685d545183688ebf37474dcec6aa64369857
Author: Gerry Kopec <gerry.kopec@windriver.com>
Date:   Thu Mar 26 00:29:58 2020 -0400

Remove dcorch-snmp and move alarm audit to dcmanager
    
    The current implementation of alarm aggregation sets a snmp trapdest on
    all subclouds to signal the system controller that the alarm status of
    the subcloud has changed.  This trap is handled by the dcorch-snmp
    process which will then trigger a rpc request to dcorch-engine which
    will in turn request an alarm summary update from the subcloud.  While
    this implementation has a throttling mechanism, it has been observed to
    bog down dorch-engine performance as the number of subclouds is
    increased.
    
    To improve scalability:
    - eliminate the dcorch-snmp process and associated subcloud trapdest
    - move existing alarm aggregation handling from dcorch to dcmanager
      process along with the db table.  Alarm status will be gathered by
      dcmanager-manager process as part of the subcloud audit.
    - remove periodic alarm audit
    - remove alarm rpc client
    - change frequent alarm related info logs to debug
    
    Test cases:
    - install dc system with multiple subclouds
    - add/manage subclouds
    - add alarms of various severity to subclouds
    - run alarm summary cli command and test alarm rest api calls
    - unmanage/poweroff/delete subclouds
    
    Change-Id: I8bd6c89f0b8a544e4d394af5bbbb86599a63823e
    Story: 2007267
    Task: 38757
    Signed-off-by: Gerry Kopec <gerry.kopec@windriver.com>

commit bd0d0c54e4c5d6fad368970622ecbb6b356648aa
Author: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date:   Wed Apr 1 19:29:16 2020 +0300

Improve handling of token expiry
    
    Extract common code.
    
    Reauthenticate ks_client.
    
    Retry once when Unauthorized, using the correct exception.
      dbs_client <-> dbsync_exceptions.Unauthorized
      ks_client <-> keystone_exceptions.Unauthorized
    
    Partial-Bug: 1862946
    
    Change-Id: Ife84b84a26ce9b82c2d9e9ca140f2a5fac000b7c
    Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>

tags:

added: in-f-centos8