Comment 3 for bug 1075901

Revision history for this message
Danny Guinther (dannyguinther) wrote :

Given that this bug has already been marked in progress, this may not be helpful, but I think this bug is more serious than suggested and should be made a priority to fix for precise.

I've run into situations where rsyslog will run at 100% CPU with no useful output or logging to indicate why. Attaching strace to the process revealed the application was spinning with the following:

futex(0x151f8e0, FUTEX_WAKE_PRIVATE, 1) = 0
open("/var/spool/rsyslog/relp_queue_hostname.00000001", O_WRONLY|O_CREAT|O_NOCTTY|O_CLOEXEC, 0600) = -1 EACCES (Permission denied)

In this particular case the problem arose because the remote relp server was down long enough to fill the local rsyslog in-memory queue and force rsyslog to spool to disk for relp. However, I have also run into situations where rsyslog will try to open a spool for the local logging queue and also run into permissions issues. In either case, if one of the queues turns to spooling, the write rate of the other queue is drastically affected, which is possibly why I wasn't getting any useful logging information from rsyslog once the spinning started.

Up until yesterday I took the approach of restarting the rsyslog process to remedy the issue which I now understand worked because it cleared the in-memory queue and thereby postponed further throttling.

This morning I forced the issue by using logger to flood rsyslog (`for i in $(seq 1 5000); do logger "Testing $i"; done`). Within a few thousand messages the in-memory queue must have reached its limit, as rsyslog began to consume 100% cpu and messages only showed up in syslog at a rate of 5-10 every minute. strace again verified rsyslog was spinning on the permissions error above.

With rsyslog in this state I issued a `sudo chown syslog:adm /var/spool/rsyslog` command and immediately cpu usage dropped to expected levels and the remaining log messages rapidly appeared in syslog (thousands per minute). With permissions correct I tried flooding rsyslog again and this time messages went directly to syslog with no adverse impact on cpu usage.

I am using precise with rsyslog 5.8.6-1ubuntu8.1.

Let me know if there's any other info I can provide or anything else I can do to help.