ceilometer swift module is spamming rsyslog with useless logs

[Joe Gordon]

Mar 18 23:21:25 devstack-precise-rax-dfw-2836361 ceilometer.openstack.common.rpc.amqp: Pool creating new connection
Mar 18 23:21:25 devstack-precise-rax-dfw-2836361 proxy-server: STDOUT: 2014-03-18 23:21:25.812 27814 DEBUG amqp [-] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2011 VMware, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'2.7.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US'] _start /usr/local/lib/python2.7/dist-packages/amqp/connection.py:751 (txn: txaca0b3d9551941c1a994d-005328d4f5)
Mar 18 23:21:25 devstack-precise-rax-dfw-2836361 proxy-server: STDOUT: 2014-03-18 23:21:25.813 27814 DEBUG amqp [-] Open OK! _open_ok /usr/local/lib/python2.7/dist-packages/amqp/connection.py:637 (txn: txaca0b3d9551941c1a994d-005328d4f5)
Mar 18 23:21:25 devstack-precise-rax-dfw-2836361 proxy-server: STDOUT: 2014-03-18 23:21:25.813 27814 DEBUG amqp [-] using channel_id: 1 __init__ /usr/local/lib/python2.7/dist-packages/amqp/channel.py:80 (txn: txaca0b3d9551941c1a994d-005328d4f5)
Mar 18 23:21:25 devstack-precise-rax-dfw-2836361 proxy-server: STDOUT: 2014-03-18 23:21:25.814 27814 DEBUG amqp [-] Channel open _open_ok /usr/local/lib/python2.7/dist-packages/amqp/channel.py:438 (txn: txaca0b3d9551941c1a994d-005328d4f5)
Mar 18

http://logs.openstack.org/32/81332/2/check/check-tempest-dsvm-neutron/cab7fc5/logs/syslog.txt.gz

In the last 12 hours abou 30% of the logs in logstash.openstack.org are from syslog which is currently being spammed by the ceilometer swift plugin

[Sean Dague (sdague)]

This is considered a critical issue because it has impacted our ability to have the logstash service available for projects.

[Nadya Privalova (nprivalova)]

The problem here is that the ceilometer-code works inside swift. So all swift logging configs should be applied to "inner"-ceilometer code. It is done here https://github.com/openstack/ceilometer/blob/master/ceilometer/objectstore/swift_middleware.py#L73 .So I think that the problem is in swift-log-configuration. Swift writes in two log-files: screen-s-proxy and syslog. It's wrong behaviour I guess but it's not a ceilometer problem. To be discussed anyway

[Nadya Privalova (nprivalova)]

Please see https://review.openstack.org/#/c/82131/1. It is in progress though

[gordon chung (chungg)]

Nadya, let's try setting Ceilometer's log level.

ie.
here: https://github.com/openstack/ceilometer/blob/master/ceilometer/objectstore/swift_middleware.py#L73
we can add: self.logger.logger.setLevel(logging.WARNING)

that way we should only see logs for warning and above related to ceilometer i think.

[Nadya Privalova (nprivalova)]

Gordon, I think we may add this here but actually it doesn't fix the problem. We just hardcode the level and doesn't make it configurable. Now the code behaves as expected: in gating we set DEBUG lvl for all services and Ceilometer middleware get all logging config from swift and just apply it. We see so much DEBUG entries because of Ceilometer's architecture: we start "mini-pipeline" in swift and get all debug messages from publishers...
To completely resolve the problem we should:
1. Add syslog-config to swift (== add oslo.logging support)
2. Think about ceilometer-middleware architecture, do we really need to start the whole pipeline?

But I agree that the items above is mostly for J. So I'm ok to hardcode the level to quick resolve the bug.

[Chuck Thier (cthier)]

I'm still trying to look into this, but you can set the log level in the config as well with something like

log_level = ERROR

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/82526

[gordon chung (chungg)]

Nadya, agreed, your plans are too grand :)

i don't see an issue with 'hardcoding' the level... we do it in ceilometer as well to hide log levels which are from external libraries which are not ceilometer specific (https://github.com/openstack/ceilometer/blob/master/ceilometer/service.py#L112). setting the logger level in middleware is similar albeit we set it on ceilometer side.

i guess to make it configurable you could add a param to filter to change levels. let me know if you have any questions on how to implement that.

[Chuck Thier (cthier)]

Since you are using swift's get_logger, it will use whatever is set with the log_level item to set the log level.

[Chuck Thier (cthier)]

I'm setting to invalid for swift as I have worked with gordc in #openstack-ceilometer oh how to set the right config to change the log level for just the ceilometer middleware in the config, needing no code change, and just an update to devstack.

[gordon chung (chungg)]

adding devstack as affected project. Nadya, please see comments i made to https://review.openstack.org/#/c/82526/

thanks for the help Chuck

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/102216

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104545

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104545
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=385bdbafc1604ac0a3f07f3e469b19bf80e51145
Submitter: Jenkins
Branch: master

commit 385bdbafc1604ac0a3f07f3e469b19bf80e51145
Author: eNovance <email address hidden>
Date: Thu Jul 3 15:06:08 2014 +0200

    update default_log_levels set by ceilometer

    the current log levels are pretty old. update the current list to
    extend current default default_log_level defined in oslo-incubator
    and adding in stevedore and keystoneclient levels

    Change-Id: I239645ef641613a994a48f20c0238e76dc9660f2
    Partial-Bug: #1294789

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/102216
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=eedfdee060b17d38b14628b7e706967cb15a35af
Submitter: Jenkins
Branch: master

commit eedfdee060b17d38b14628b7e706967cb15a35af
Author: Dina Belova <email address hidden>
Date: Tue Jun 24 16:52:46 2014 +0400

    Fix Swift config for Ceilometer usage

    Ceilometer filter was previously removed from the Swift
    configuration to avoid useless Ceilometer logs in the Swift
    ones. This was fixed by setting only warning log level for
    this Ceilometer part of the pipeline to keep only important
    ones.

    Change-Id: I8c41355bb98dbf3bb59ec792221b05ea936086b7
    Partial-Bug: #1294789

[gordon chung (chungg)]

marking this part of logging issue complete. ceilometer has been updated to include default log levels typically assigned to third party libraries... future ceilometer specific logging issues will be addressed by bp: https://review.openstack.org/#/c/97261/

[gordon chung (chungg)]

marking logging issue regarding syslog complete as of Dina's patch

[OpenStack Infra (hudson-openstack)]

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/82526
Reason: this has been addressed... closing patch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/116229

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/116230

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/116229
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=014c41ee41f0afa12646d88d47fba70c4d2ce368
Submitter: Jenkins
Branch: stable/icehouse

commit 014c41ee41f0afa12646d88d47fba70c4d2ce368
Author: eNovance <email address hidden>
Date: Thu Jul 3 15:06:08 2014 +0200

    update default_log_levels set by ceilometer

    the current log levels are pretty old. update the current list to
    extend current default default_log_level defined in oslo-incubator
    and adding in stevedore and keystoneclient levels

    that is needed in stable/icehouse to allow swift+ceilometer
    in devstack and devstack gate (otherwise grenade job, that is
    using stable/icehouse devstack, won't pass the gate)

    Change-Id: I239645ef641613a994a48f20c0238e76dc9660f2
    Partial-Bug: #1294789
    (cherry picked from commit 4d864cef5a2c8ec912bda5b208771d3b7c966934)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/116230
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=0b0f1fa3c773bef64bc2ce46fd58575ef84f9e81
Submitter: Jenkins
Branch: stable/icehouse

commit 0b0f1fa3c773bef64bc2ce46fd58575ef84f9e81
Author: Dina Belova <email address hidden>
Date: Tue Jun 24 16:52:46 2014 +0400

    Fix Swift config for Ceilometer usage

    Ceilometer filter was previously removed from the Swift
    configuration to avoid useless Ceilometer logs in the Swift
    ones. This was fixed by setting only warning log level for
    this Ceilometer part of the pipeline to keep only important
    ones.

    that is needed in stable/icehouse to allow swift+ceilometer
    in devstack and devstack gate (otherwise grenade job, that is
    using stable/icehouse devstack, won't pass the gate)

    Change-Id: I8c41355bb98dbf3bb59ec792221b05ea936086b7
    Partial-Bug: #1294789
    (cherry picked from commit eedfdee060b17d38b14628b7e706967cb15a35af)

[Sean Dague (sdague)]

This devstack bug was last updated over 180 days ago, as devstack
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.