Bug in syslogd-listfiles causing insane log rotation

[Thomas Guyot-Sionnest]

Binary package hint: sysklogd

** First some background information:

About a year ago I enabled syslog to listen on my private LAN so I could forward syslog message from my OpenWTR (linux) NAT router and save it on my workstation (to debug an intermittent problem with the ADSL PPPoE handshake). To make sure I'd get all logs from the router in a single place I added the following line in syslog.conf:

  *.* /var/log/everything.log

** The first side effects of the bug:

A few months ago I started having login issues, both with local (console and GUI) login and ssh. I added a bare root shell (no login) on tty12 to debug when it happens and I finally found out timeouts were occurring while reading /var/log. It turned out I had millions of files with a few random .0, .2, .3, .4 or .gz extentions and logins were timing out while trying to read that folder (probably somewhere in the pam/logging code.

I knew it was something with the log rotation scripts but didn't had time to find the culprit until now. I was working around it with a periodic:

rm -f /var/log/*.{[0-9],gz}.{[0-9],gz}

** The root cause:

It turns out /etc/cron.weekly/sysklogd runs '/usr/sbin/syslogd-listfiles --weekly' to get its list of log files and the above syslog.conf line cause it to return '*.*' among the list!

I quickly looked at /usr/sbin/syslogd-listfiles without trying much to find the proper fix, but I also noted that comments starting with a whitespace before the # are also included in the files list.

Given the impact this bug can have on the system (causing login timeouts) and the little work required to fix it I believe it should be treated as urgent.

[Daniel T Chen (crimsun)]

Is this symptom still reproducible in 8.10 or 9.04?

[Thomas Guyot-Sionnest (dermoth)]

I could test once I have time to upgrade to 8.10; in the mean time I started logging to a named pipe to avoid the bug,

You could easily test yourself by adding that line to your syslog.conf, or even running the aforementioned function on a dummy file with that line.

Or you could point me to a place where I can get /usr/sbin/syslogd-listfiles from 8.10 or 9.04 (or attach them to this bug) so I can test myself.

I'm stunned that you set this to low priority though... This bug cause the log rotation script to create thousand of files in /var/log as long as you log everything (*.*) to a regular file and the end result over time is having login take very long time to run; in my case even timing out logins multiple times until enough directory entries could end up in the system cache.

[Ralph Janke (txwikinger)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue for you. Can you try with the latest Ubuntu release? Thanks in advance.

[Thomas Guyot-Sionnest (dermoth)]

I'm currently using Hardy and I can still reproduce it. You should be able to test easily with these steps:

As root, run this (or edit syslog.conf accordingly):

# echo -e "\t*.*\t\t\t/var/log/everything.log" >>/etc/syslog.conf

Run:

$ /usr/sbin/syslogd-listfiles --weekly

Is the command returns *.* as a log file then the problem is still present.

This is pretty nasty because this cause hard to catch problems and the root cause is not evident either. This is very likely an error in the regexp syslogd-listfiles use but I haven't looked at it yet.

[Thomas Guyot-Sionnest (dermoth)]

It not fixed in the latest branch code (http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/sysklogd/lucid/files/head:/debian/), but with the move to rsyslog it doesn't seem to be a problem anymore on default setups.

Steps to reproduce:

1. Download syslogs-listfiles

Since I upgraded to lucid and it uses rsyslog I downloaded it straight from the branch:
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/sysklogd/lucid/download/head%3A/syslogdlistfiles-20090626232137-tixv5m2ox9we304t-56/syslogd-listfiles

2. add this line (written in c-format) to /etc/syslogd.conf:

\t*.*\t\t\t/var/log/everything.log

ex., run as root:
# echo -e "\t*.*\t\t\t/var/log/everything.log" >>/etc/syslog.conf

3. run the script with the weekly argument:

$ /usr/sbin/syslogd-listfiles --weekly

Result:
The command returns *.*, which matches also rotated logs and cause uncontrollable growth of rotated logs

Expected result:
Should return only /var/log/everything.log

Additional notes:
I cannot find a way to the the file returned with *.*, there must be some facility or priority match. The line is ignored if there is no leading tab, but sysklogd ignores leading tabs.