wsgi and eventlet logs flooding on large deployments

[Kam Nasim]

Based on observation in one of our larger lab deployments, keystone/keystone-all.log is seeing on average multiple logs per seconds. These are logged at INFO verbosity which is the default logging level in keystone.conf

Most frequently seen logs:
<DATE> <TIME> <NUMBER> INFO keystone.common.wsgi [-] GET /auth/tokens?

<DATE> <TIME> <NUMBER> INFO eventlet.wsgi.server [-] <NUMBER> - - [<DATE> <TIME>] "GET /v <NUMBER> /auth/tokens HTTP/ <NUMBER> " <NUMBER> . <NUMBER>

Other high runner logs:
<DATE> <TIME> <NUMBER> INFO keystone.common.wsgi [-] POST /tokens?POST /tokens

<DATE> <TIME> <NUMBER> INFO eventlet.wsgi.server [-] <NUMBER> - - [<DATE> <TIME>] "POST /v <NUMBER> /tokens HTTP/ <NUMBER> " <NUMBER> . <NUMBER>

<DATE> <TIME> <NUMBER> INFO keystone.common.wsgi [-] GET /?
<DATE> <TIME> <NUMBER> INFO eventlet.wsgi.server [-] <NUMBER> - - [<DATE> <TIME>] "GET / HTTP/ <NUMBER> " <NUMBER> <NUMBER>

These log entries cause frequent churn in keystone-all.log.

We have a small fix in place to move GET/POST logs from keystone.common.wsgi and eventlet.wsgi.server to debug logs and to enable these logs when the verbosity is set to debug in keystone.conf

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/329028

[Guang Yee (guang-yee)]

You can also control the log level per module here

https://github.com/openstack/keystone/blob/master/etc/keystone.conf.sample#L190

or here

https://github.com/openstack/keystone/blob/master/etc/keystone.conf.sample#L132

For the latter, it should be similar to logging.conf (https://docs.python.org/2/library/logging.config.html).

[Steve Martinelli (stevemar)]

guang has a point, but it may be worth to change the default value to debug so deployers don't have to

[Steve Martinelli (stevemar)]

Is this specific to eventlet or keystone in general? eventlet is removed in the newton release.

[Kam Nasim (knasim-wrs)]

the original fix that we had in our deployment was for both eventlet and wsgi. However since eventlets were removed in HEAD around March year, the Upstream port is for wsgi logging only.

Given that the review is split 50/50 with half the reviewers in favour of keeping the log verbosity as is, I will withdraw the inspection

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kam Nasim (<email address hidden>) on branch: master
Review: https://review.openstack.org/329028
Reason: withdrawing this commit as per review feedback. The debug verbosity will be kept at present level