Neutron log obesity epidepmic

[Kevin Benton]

From a single scenario job (http://logs.openstack.org/57/488557/1/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/36d8e0b/logs/?C=S;O=D)

[ ] screen-q-agt.txt.gz 2017-07-28 19:45 9.6M
[ ] screen-q-svc.txt.gz 2017-07-28 19:45 7.1M

Our compressed log sizes are 9x for server and 7x for agent the size of the next in the list (Keystone). Before the release I would like to trim down on the debug messages.

[Brian Haley (brian-haley)]

From the file you linked, two-thirds of the lines are from ovsdbapp:

--> wc -l screen-q-agt.txt
368025 screen-q-agt.txt

--> grep -c ovsdbapp screen-q-agt.txt
214062

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/489428

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/489429

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/489430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/489430
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d92dc1244ed8277bf0c204448b49a425b40f55b1
Submitter: Jenkins
Branch: master

commit d92dc1244ed8277bf0c204448b49a425b40f55b1
Author: Kevin Benton <email address hidden>
Date: Mon Jul 31 15:54:20 2017 -0700

    Change ovsdbapp vlog to INFO

    This is to eliminate the following messages that fill the
    OVS agent logs:

    ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN]

    Change-Id: I00010ef600a4185b7628318bb88971d978b9fe67
    Partial-bug: #1707307

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/489428
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ce8dbcff838a6e89cba87882d84365db590508b8
Submitter: Jenkins
Branch: master

commit ce8dbcff838a6e89cba87882d84365db590508b8
Author: Kevin Benton <email address hidden>
Date: Mon Jul 31 15:46:39 2017 -0700

    Don't log 'Exit code: 0'

    A regular gate run shows this was logged ~2500 times.
    It doesn't provide any useful information since the
    command executed successfully and we can see the
    issued command in the debug statement before it.

    Partial-Bug: #1707307
    Change-Id: I431d6a59ce36deb9a5fb9deef39655d28d71f24d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/490212

[OpenStack Infra (hudson-openstack)]

Change abandoned by Miguel Lavalle (<email address hidden>) on branch: master
Review: https://review.openstack.org/490212

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/490213

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/492418

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/492419

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/492420

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/489429
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=72f7f1eaa8a4b9b2c792d4466506b8164020d922
Submitter: Jenkins
Branch: master

commit 72f7f1eaa8a4b9b2c792d4466506b8164020d922
Author: Kevin Benton <email address hidden>
Date: Mon Jul 31 15:51:57 2017 -0700

    Drop port_delete and port_update debug msgs

    The push notification resource cache will already give us
    the port update and delete notifications so there is no need
    to log here now.

    Change-Id: I6000691dfd61facf47d70360d4050cfdefc3e7e2
    Partial-Bug: #1707307

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/490213
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1865f27a8b98f322caaad46cbb145c0972753d84
Submitter: Jenkins
Branch: master

commit 1865f27a8b98f322caaad46cbb145c0972753d84
Author: Miguel Lavalle <email address hidden>
Date: Wed Aug 2 16:34:27 2017 -0500

    Remove code to debug auto address allocation error

    Change If1eb4046865f43b15ba97c52e2d0b9343dc72c19 fixed bugs 1666493 and 1655567
    that reported exception IpAddressAlreadyAllocated being raised during the
    creation of IPv6 auto-address subnets. This patchset removes code that was
    added by change I22b8f1f537f905f4b82ce9e50d6fcc5bf2210f9f to root-cause these
    bugs.

    By removing log statements, this patchset contributes to reduce the
    'Neutron log obesity epidemic'

    Change-Id: I28c58dc4a957df833d277f0d08ce831c7ee07c68
    Partial-Bug: #1707307

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492419
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c36676877eccca983aba1bf64a953bc65bf78c36
Submitter: Jenkins
Branch: master

commit c36676877eccca983aba1bf64a953bc65bf78c36
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:13:21 2017 -0700

    Remove 'persisted dirty' log message

    In a gate run the neutron.quota.resource module logged 5235
    lines from this module, making it one of the top offenders.
    This single line accounted for 3297 of those entries and it
    offers limited information since the quota engine is quite
    stable at this point and debugging tenants' dirty status isn't
    an issue that comes up.

    We can just get rid of this particular log line.

    Change-Id: I9cb0fcfbad51a0863d04fb89fbb8b09e3e21bdf0
    Partial-Bug: #1707307

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492420
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2d8ffe2a08120f835c875060a197b4db2ec8c735
Submitter: Jenkins
Branch: master

commit 2d8ffe2a08120f835c875060a197b4db2ec8c735
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:29:43 2017 -0700

    Stop logging full object in RPC push code

    We already log the full object on the agent side
    and other server methods log important parts of
    the object so there isn't much to gain logging the full
    OVO components passed to this function's args.

    This adds a specific debug statement that just indicates
    the types, IDs, and revision_numbers being pushed out so
    they can still be correlated with received objects on the
    agent side.

    Partial-Bug: #1707307
    Change-Id: I4499bb328f0aeb58fe583b83fb42cd2d26c1c4c1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492418
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=62c1a4b6873d0b2382349ece65545bf28c9c87ca
Submitter: Jenkins
Branch: master

commit 62c1a4b6873d0b2382349ece65545bf28c9c87ca
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:07:31 2017 -0700

    Log policy filters in one line

    Rather than generating an entire log line for each policy that
    filters out an attribute, we can instead log the excluded attributes
    once after all of the policy checks are done.

    This only applies to the policy checks, policy enforcements
    (e.g. preventing a user from entering a field value on create)
    are still logged individually.

    Partial-Bug: #1707307
    Change-Id: I77401e30cb220901dacd61028acde4c79001ed9b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/496927

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492421
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=03898582c81547f048fd9c1d51527c7f6c44f6ba
Submitter: Jenkins
Branch: master

commit 03898582c81547f048fd9c1d51527c7f6c44f6ba
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:51:00 2017 -0700

    Stop logging versions on every agent update

    This logging statement was being called on every agent update
    even if no versions changed, resulting in a bunch of useless
    messages.

    We already log when the actual versions change in the _set_version
    call here: [1]

    To cover the cases where versions are removed, this adds two logging
    statements so we can be sure we see when the tracked versions do
    change but not be bothered by version logs in any other condition.

    1.
    http://git.openstack.org/cgit/openstack/neutron/tree/neutron/api/rpc/callbacks/version_manager.py?h=d225b86738a9db56bc643dfe637b1f21393ad27e#n113

    Partial-Bug: #1707307
    Change-Id: I4761d21257dabfbd661bba1b24fe09d208d16ae6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/498000

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/498001

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/498002

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/496927
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=de712bf467888b05de14bcbe6e65cc1765692c62
Submitter: Jenkins
Branch: stable/pike

commit de712bf467888b05de14bcbe6e65cc1765692c62
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:07:31 2017 -0700

    Log policy filters in one line

    Rather than generating an entire log line for each policy that
    filters out an attribute, we can instead log the excluded attributes
    once after all of the policy checks are done.

    This only applies to the policy checks, policy enforcements
    (e.g. preventing a user from entering a field value on create)
    are still logged individually.

    Partial-Bug: #1707307
    Change-Id: I77401e30cb220901dacd61028acde4c79001ed9b
    (cherry picked from commit 62c1a4b6873d0b2382349ece65545bf28c9c87ca)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/498002
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=406472f3ea5aebbb6d02763cf21da4488095daed
Submitter: Jenkins
Branch: stable/newton

commit 406472f3ea5aebbb6d02763cf21da4488095daed
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:51:00 2017 -0700

    Stop logging versions on every agent update

    This logging statement was being called on every agent update
    even if no versions changed, resulting in a bunch of useless
    messages.

    We already log when the actual versions change in the _set_version
    call here: [1]

    To cover the cases where versions are removed, this adds two logging
    statements so we can be sure we see when the tracked versions do
    change but not be bothered by version logs in any other condition.

    1.
    http://git.openstack.org/cgit/openstack/neutron/tree/neutron/api/rpc/callbacks/version_manager.py?h=d225b86738a9db56bc643dfe637b1f21393ad27e#n113

    Partial-Bug: #1707307
    Change-Id: I4761d21257dabfbd661bba1b24fe09d208d16ae6
    (cherry picked from commit 03898582c81547f048fd9c1d51527c7f6c44f6ba)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/498000
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=38a98fcfed5c40bf031fe1cc96359a819dc4953c
Submitter: Jenkins
Branch: stable/pike

commit 38a98fcfed5c40bf031fe1cc96359a819dc4953c
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:51:00 2017 -0700

    Stop logging versions on every agent update

    This logging statement was being called on every agent update
    even if no versions changed, resulting in a bunch of useless
    messages.

    We already log when the actual versions change in the _set_version
    call here: [1]

    To cover the cases where versions are removed, this adds two logging
    statements so we can be sure we see when the tracked versions do
    change but not be bothered by version logs in any other condition.

    1.
    http://git.openstack.org/cgit/openstack/neutron/tree/neutron/api/rpc/callbacks/version_manager.py?h=d225b86738a9db56bc643dfe637b1f21393ad27e#n113

    Partial-Bug: #1707307
    Change-Id: I4761d21257dabfbd661bba1b24fe09d208d16ae6
    (cherry picked from commit 03898582c81547f048fd9c1d51527c7f6c44f6ba)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/498001
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1c6d8b255baea287489db8a36c339798ddc1b44f
Submitter: Jenkins
Branch: stable/ocata

commit 1c6d8b255baea287489db8a36c339798ddc1b44f
Author: Kevin Benton <email address hidden>
Date: Thu Aug 10 00:51:00 2017 -0700

    Stop logging versions on every agent update

    This logging statement was being called on every agent update
    even if no versions changed, resulting in a bunch of useless
    messages.

    We already log when the actual versions change in the _set_version
    call here: [1]

    To cover the cases where versions are removed, this adds two logging
    statements so we can be sure we see when the tracked versions do
    change but not be bothered by version logs in any other condition.

    1.
    http://git.openstack.org/cgit/openstack/neutron/tree/neutron/api/rpc/callbacks/version_manager.py?h=d225b86738a9db56bc643dfe637b1f21393ad27e#n113

    Partial-Bug: #1707307
    Change-Id: I4761d21257dabfbd661bba1b24fe09d208d16ae6
    (cherry picked from commit 03898582c81547f048fd9c1d51527c7f6c44f6ba)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/499533

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499533
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=22a63315d85db05eaa67f3599caa82c5966b880c
Submitter: Jenkins
Branch: stable/pike

commit 22a63315d85db05eaa67f3599caa82c5966b880c
Author: Miguel Lavalle <email address hidden>
Date: Wed Aug 2 16:34:27 2017 -0500

    Remove code to debug auto address allocation error

    Change If1eb4046865f43b15ba97c52e2d0b9343dc72c19 fixed bugs 1666493 and 1655567
    that reported exception IpAddressAlreadyAllocated being raised during the
    creation of IPv6 auto-address subnets. This patchset removes code that was
    added by change I22b8f1f537f905f4b82ce9e50d6fcc5bf2210f9f to root-cause these
    bugs.

    By removing log statements, this patchset contributes to reduce the
    'Neutron log obesity epidemic'

    Change-Id: I28c58dc4a957df833d277f0d08ce831c7ee07c68
    Partial-Bug: #1707307

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/505384

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505384
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f8fb73aa30f075712b5ca18c78b88d6ae9e60566
Submitter: Jenkins
Branch: master

commit f8fb73aa30f075712b5ca18c78b88d6ae9e60566
Author: Kevin Benton <email address hidden>
Date: Tue Sep 19 12:18:48 2017 -0700

    Dont log about skipping notification in normal case

    Avoid logging about a missing resource for normal cases
    where there is no resource in the request (e.g. HTTP GET's,
    custom actions, etc).

    Change-Id: I04c6b6737f238ca6250869b92e1d0771b4f95f22
    Partial-Bug: #1707307

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/506892

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506892
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4cbf3153e840315daae698037ec9c6a785d4e5d8
Submitter: Jenkins
Branch: stable/pike

commit 4cbf3153e840315daae698037ec9c6a785d4e5d8
Author: Kevin Benton <email address hidden>
Date: Tue Sep 19 12:18:48 2017 -0700

    Dont log about skipping notification in normal case

    Avoid logging about a missing resource for normal cases
    where there is no resource in the request (e.g. HTTP GET's,
    custom actions, etc).

    Change-Id: I04c6b6737f238ca6250869b92e1d0771b4f95f22
    Partial-Bug: #1707307
    (cherry picked from commit f8fb73aa30f075712b5ca18c78b88d6ae9e60566)