Distributed Cloud: unexpected network status out of sync events flooding logs

Bug #1802397 reported by Maria Yousaf
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Kristine Bujold

Bug Description

Title
-----
Distributed cloud - unexpected network status out of sync events flooding logs

Brief Description
-----------------
I'm seeing network out-of-sync events being reported against all the subclouds in the distributed cloud system. The subclouds are reported as being in-sync according to the Cloud Overview. The audit runs and reports all of them out-of-sync from a network resource perspective. When the audit is done, they report in-sync again. This was observed even on subclouds that would have been idle overnight. This behaviour happens every audit cycle. The end effect is that the logs are flooded with these alarms.

Severity
--------
Major

Steps to Reproduce
------------------
Install a distributed cloud system

Expected Behavior
------------------
Network out of sync alarms not seen

Actual Behavior
----------------
Network out of sync alarms are being seen

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Distributed cloud

Branch/Pull Time/Commit
-----------------------

Timestamp/Logs
--------------

Revision history for this message
Ghada Khalil (gkhalil) wrote :

Targeting stx.2019.03 - log flooding which can impact debug-ability. should be addressed before the release date.

summary: - Distributed cloud - unexpected network status out of sync events
- flooding logs
+ Distributed Cloud: unexpected network status out of sync events flooding
+ logs
Changed in starlingx:
assignee: nobody → Kristine Bujold (kbujold)
importance: Undecided → Medium
status: New → Triaged
tags: added: stx.2019.03 stx.distcloud
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-distcloud (master)

Reviewed: https://review.openstack.org/616980
Committed: https://git.openstack.org/cgit/openstack/stx-distcloud/commit/?id=7c1c082be0e9c9bacfd251fe3fa1de33a0d2864f
Submitter: Zuul
Branch: master

commit 7c1c082be0e9c9bacfd251fe3fa1de33a0d2864f
Author: Kristine Bujold <email address hidden>
Date: Fri Nov 9 10:23:56 2018 -0500

    DC fix audit sync failure

    During the sync Security Group needs to be compared with the tenant
    names and not tenant ids. The tenant id which is the uuid are different
    between regions. The compare does not match and an attempt to create
    the Security Group fails since it already exists. This commit changes
    the compare to use tenant names.

    Closes-Bug: 1802397

    Change-Id: I9363945868b857802e137dcf05757ec3691230c1
    Signed-off-by: Kristine Bujold <email address hidden>

Changed in starlingx:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to stx-distcloud (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/620701

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to stx-distcloud (master)

Reviewed: https://review.openstack.org/620701
Committed: https://git.openstack.org/cgit/openstack/stx-distcloud/commit/?id=149979407eb2c3c1720a5b20ff66e0543ebd04e7
Submitter: Zuul
Branch: master

commit 149979407eb2c3c1720a5b20ff66e0543ebd04e7
Author: Kristine Bujold <email address hidden>
Date: Wed Nov 28 16:15:21 2018 -0500

    Adding TODO comment

    Adding a clarification to earlier commit 7c1c082b.

    Related-Bug: 1802397

    Change-Id: I9c008db783fa32c5747e416844f76315bd4dc3ba
    Signed-off-by: Kristine Bujold <email address hidden>

Ken Young (kenyis)
tags: added: stx.2019.05
removed: stx.2019.03
Ken Young (kenyis)
tags: added: stx.2.0
removed: stx.2019.05
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.