new exception summary polluting logs on retry

Bug #1696855 reported by Kevin Benton
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
oslo.log
Invalid
Medium
Doug Hellmann

Bug Description

We extensively use the olso.db retry decorator[1], which will catch retriable exceptions (e.g. Staledataerrors, DBDeadlocks, etc) and recall the function. Either because of the loop holding a reference to the exception, or one of our outer wrappers inspecting the exception before reraising[2], the new exception summaries added in [3] result in every logging statement during the retry having irrelevant exception info added.

For an example, this line[4] is logging a StaleDataError that triggered the retry. However, that is completely irrelevant to the ongoing operation (we already log that something is being retried because of a retriable exception).

We need a way to get this to stop because it makes the logs really messy whenever there is a retry.

1. https://github.com/openstack/oslo.db/blob/master/oslo_db/api.py#L130
2. https://github.com/openstack/neutron/blob/master/neutron/db/api.py#L103
3. https://github.com/openstack/oslo.log/commit/48d284a9856dbaeb2890154bb71abc55dd0bee7a
4. http://logs.openstack.org/59/471059/2/check/gate-neutron-dsvm-fullstack-ubuntu-xenial/df11b90/logs/dsvm-fullstack-logs/TestHAL3Agent.test_keepalived_multiple_sighups_does_not_forfeit_mastership/neutron-server--2017-06-05--21-41-34-957535.txt.gz#_2017-06-05_21_42_05_746

Changed in oslo.log:
importance: Undecided → Medium
status: New → Confirmed
assignee: nobody → Doug Hellmann (doug-hellmann)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to oslo.log (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/473611

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to oslo.log (master)

Reviewed: https://review.openstack.org/473611
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=527a76e4e33a980c925315dc7700afcb060c365c
Submitter: Jenkins
Branch: master

commit 527a76e4e33a980c925315dc7700afcb060c365c
Author: Doug Hellmann <email address hidden>
Date: Mon Jun 12 18:05:31 2017 -0400

    do not add error_summary for debug log messages

    We catch exceptions and log debug messages a lot, which results in extra
    occurrences of the exception when we don't need them.

    Change-Id: I4dd3bf65fb456afc07a8fa7d0c9869bcc266b800
    Related-Bug: #1696213
    Related-Bug: #1696855
    Signed-off-by: Doug Hellmann <email address hidden>

Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

Is this still a problem after the change in #2?

Changed in oslo.log:
status: Confirmed → Incomplete
Changed in oslo.log:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.