Sub clouds going offline and losing sync with the System Controller regularly
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
High
|
Tao Liu |
Bug Description
Brief Description
-----------------
Sub clouds are going offline and losing the sync with System Controller due to the following error: "No auth_url provided: HTTPUnauthorized: No auth_url provided".
Severity
--------
Major.
Steps to Reproduce
------------------
1. Bootstrap and Manage the sub clouds.
2. Monitor the sub clouds using "dcmanager subcloud list" command and verify that the sub clouds go offline and lose the sync with System Controller continuously.
Expected Behavior
------------------
Sub clouds online/in-sync during all the time they are up and running.
Actual Behavior
----------------
Sub clouds that are online/in-sync suddenly go to offline/
Reproducibility
---------------
Intermittent issue. About 10% of the sub clouds are always going offline and losing the sync.
System Configuration
-------
Distributed Cloud with sub clouds deployed on AWS.
Branch/Pull Time/Commit
-------
N/A.
Last Pass
---------
N/A.
Timestamp/Logs
--------------
2021-04-19 17:12:53.135 115026 ERROR dccommon.
2021-04-19 17:12:53.138 115026 ERROR dcmanager.
2021-04-19 17:12:53.139 115026 INFO dcmanager.
2021-04-19 17:12:54.131 115026 INFO dcmanager.
2021-04-19 17:13:25.050 115026 INFO dcmanager.
2021-04-19 17:13:25.262 115026 INFO dcmanager.
2021-04-19 17:13:25.638 115026 INFO dcmanager.
2021-04-19 17:13:25.777 115026 INFO dccommon.
Test Activity
-------------
System Test.
Workaround
----------
N/A - Sub clouds are able to recover by themselves but can go offline again later.
Changed in starlingx: | |
assignee: | nobody → Tao Liu (tliu88) |
tags: | added: stx.cherrypickneeded |
tags: |
added: in-r-stx50 removed: stx.cherrypickneeded |
Fix proposed to branch: master /review. opendev. org/c/starlingx /distcloud/ +/789572
Review: https:/