Maintenance task doesn't have enough logging verbosity

Bug #1837003 reported by Daniel Alvarez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-ovn
Fix Released
Medium
Lucas Alvares Gomes

Bug Description

When the maintenance task kicks in and fixes some inconsistencies, it's hard to tell through the logs. Plus it doesn't have enough verbosity in the logs to tell which objects have been fixed.

Right now, when debugging this issue, I've been grepping for "Maintenance task synchronization finished" and scrolling back on the logs to figure out via ovsdbapp traces what have been fixed.

It'd be nice if we could add a summary of the problems detected at [0]. This will also help figuring out how likely race conditions occur and which objects are more prone to those under load.

[0] https://github.com/openstack/networking-ovn/blob/6.0.0.0rc1/networking_ovn/common/maintenance.py#L261

Changed in networking-ovn:
importance: Undecided → Medium
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (master)

Fix proposed to branch: master
Review: https://review.opendev.org/671442

Changed in networking-ovn:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (master)

Reviewed: https://review.opendev.org/671442
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=f59acf38968ce9c6129f1a1bffd277af8406a2d0
Submitter: Zuul
Branch: master

commit f59acf38968ce9c6129f1a1bffd277af8406a2d0
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 17 15:06:09 2019 +0100

    Maintenance task: Enhance debug logging

    This patch is enhancing the maintenance task debug logs by:

    * Logging a summary with the number of failures on each resource type
    * Logging the UUID and type of the resource being fixed at that time
    * Changing the prefix of the maintenance logs for easy grepping

    Closes-Bug: #1837003
    Change-Id: I0faa19ef48f2dc5591c0337ae5a0599abc2e4168
    Signed-off-by: Lucas Alvares Gomes <email address hidden>

Changed in networking-ovn:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/673000

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/673001

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/673003

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (stable/stein)

Reviewed: https://review.opendev.org/673000
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=4232a2fabdefb124f9abdfa0b399a01aba14299a
Submitter: Zuul
Branch: stable/stein

commit 4232a2fabdefb124f9abdfa0b399a01aba14299a
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 17 15:06:09 2019 +0100

    Maintenance task: Enhance debug logging

    This patch is enhancing the maintenance task debug logs by:

    * Logging a summary with the number of failures on each resource type
    * Logging the UUID and type of the resource being fixed at that time
    * Changing the prefix of the maintenance logs for easy grepping

    Conflicts:
     networking_ovn/common/maintenance.py

    Closes-Bug: #1837003
    Change-Id: I0faa19ef48f2dc5591c0337ae5a0599abc2e4168
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit f59acf38968ce9c6129f1a1bffd277af8406a2d0)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (stable/rocky)

Reviewed: https://review.opendev.org/673001
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=6570ffa52f30ebc208a4564d45fd92b389298e14
Submitter: Zuul
Branch: stable/rocky

commit 6570ffa52f30ebc208a4564d45fd92b389298e14
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 17 15:06:09 2019 +0100

    Maintenance task: Enhance debug logging

    This patch is enhancing the maintenance task debug logs by:

    * Logging a summary with the number of failures on each resource type
    * Logging the UUID and type of the resource being fixed at that time
    * Changing the prefix of the maintenance logs for easy grepping

    Conflicts:
     networking_ovn/common/maintenance.py

    Closes-Bug: #1837003
    Change-Id: I0faa19ef48f2dc5591c0337ae5a0599abc2e4168
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit f59acf38968ce9c6129f1a1bffd277af8406a2d0)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (stable/queens)

Reviewed: https://review.opendev.org/673003
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=5fed25fdef377207a2a2240d6385ae187a8ee384
Submitter: Zuul
Branch: stable/queens

commit 5fed25fdef377207a2a2240d6385ae187a8ee384
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 17 15:06:09 2019 +0100

    Maintenance task: Enhance debug logging

    This patch is enhancing the maintenance task debug logs by:

    * Logging a summary with the number of failures on each resource type
    * Logging the UUID and type of the resource being fixed at that time
    * Changing the prefix of the maintenance logs for easy grepping

    Conflicts:
     networking_ovn/common/maintenance.py

    Closes-Bug: #1837003
    Change-Id: I0faa19ef48f2dc5591c0337ae5a0599abc2e4168
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit f59acf38968ce9c6129f1a1bffd277af8406a2d0)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 7.0.0.0b1

This issue was fixed in the openstack/networking-ovn 7.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 4.0.4

This issue was fixed in the openstack/networking-ovn 4.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 6.0.1

This issue was fixed in the openstack/networking-ovn 6.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 5.1.0

This issue was fixed in the openstack/networking-ovn 5.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.