lxc container service logs filling syslog

[Surajit Roy]

We have seen a scenario in Liberty where if we have a network glitch between the compute nodes and the rabbitmq, and few nodes does not recover well - the rabbitmq container syslog starts filling with rabbitmq errors and the container disk run out of disk space and eventually the container goes down.

The workaround to that is restart all the services and cleanup syslog in the container.

[Jean-Philippe Evrard (jean-philippe-evrard)]

Please see our bug triage conversation here: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2016-09-13.log.html#t2016-09-13T16:20:31

[Surajit Roy (beasurajitroy)]

On the re-direction of logs I have been looking at rsyslog setting and one of the theories derived by me(still under more research) is
Default rsyslogd conf
*.* ;auth,authpriv.none -/var/log/syslog
*.* means log all facilities and all priorities.
auth,authpriv.none means don't log the auth and authpriv facilities.
-/var/log/syslog means log to the file /var/log/syslog. The preceding dash tells syslogd not to call fsync(), i.e. do not flush the kernel buffer to disk after every write to the file.

It could be possible that due to this setting , for a burst of messages logs are getting queued and eventually written to syslog. I looking more into the rsyslog internals to prove the theory.

We are testing it at our end.
One of the behaviors we saw was if logging is aggressive all the logs get redirected.
If we remove this option we dont see the behavior.

Logrotate is the last option if we are not able to figure out the reason of re-direction.
I would say we go with size based rotation in that case. And we can decide with the size that we want to retain.

[Surajit Roy (beasurajitroy)]

https://review.openstack.org/#/c/351182/