auth.log is empty

[nick parlante]

On a fresh 12.04 64 bit machine in the default state + sshd installed, the auth.log file remained empty, when normally it would fill up with sshd hacking attempts. The sshd_config was left at its default, which should record login failures.

I have figured out a workaround, which is probably a good clue about the underlying bug.

It turns out that the permissions of auth.log were: messagebus (owner) adm (group)

doing a

sudo chown syslog /etc/auth.log

fixed the problem instantly, with failed logins now going to the file as expected. I don't know if this "fix" will survive log rotation.

[nick parlante (nick-parlante)]

Following up on my own bug:

The logging of sudo sessions is another example of data that was not being written to auth.log

I see that the older auth.log.1 files have the messagebus/adm permissions, but do have content in them, running up to July 27th 2012. I assume these were written when the system was first set up by the vendor, system76.

I first booted this box up around Sep 24th 2012, and applied all the pending updates. I wonder if one of those updates introduced this problem.

[Peter Czanik (pczanik)]

I did a fresh install of 12.04, installed syslog-ng, updated the machine, but I could not reproduce this problem. Permissions were always correct and logs collected. It does not seem to be a syslog-ng package bug.

[nick parlante (nick-parlante)]

Thanks for looking in to it Peter. Sorry for the confusion, but I did not install syslog-ng. I left the machine with the default. Digging around, I now see that rsyslogd is the default. Syslog-ng is what poppped up in the package auto-complete when I typed "syslog" so I thought, incorrectly, that's what ubuntu was using.

I suspect the problem is some sort of installer bug or race condition here the auth.log file gets created with the wrong owner. Googling around, you can see forum threads where people have this symptom of auth.log staying empty now and then going back years.

This bug can at least document how to check for the problem and fix it:

/var/log/auth.log should be owned by "syslog" (not "messagebus"), so doing an ls -l should look like this
ls -l /var/log/auth.log
-rw-r----- 1 syslog adm 30310 Oct 2 12:17 /var/log/auth.log

If it is not owned by syslog, fix it like this:
sudo chown syslog /var/log/auth.log

I'll see if I can figure out how to submit a test case for it.

[nick parlante (nick-parlante)]

Ok, I've written a short Checkbox style test -- that much I could figure out. I'm hoping someone with permission to submit new checkbox tests can wrap this up as a checkbox job. It would be interesting to run it after each install/upgrade cycle as I suspect that's when this bug happens.

#!/bin/bash
#
# Test that the owner of auth.log is syslog
# This test is meaningless if auth.log does not exist, so the
# dependencies should run this after it has been created.

if [[ -e /var/log/auth.log && `stat -c %U /var/log/auth.log` != "syslog" ]]
then
  echo "auth.log is not owned by syslog" >&2
  exit 1
else
  exit 0
fi

[nick parlante (nick-parlante)]

Ok, I have a theory as to what is causing this bug and why it is hard to reproduce.

The /etc/logrotate.conf specifies "create" and later the logrotate.d/rsyslog lists a bunch of log files.

The behavior of "create" is that it creates the new file copying the owner etc. from the existing file. However, this creates a window in rare cases where there is no existing log file for whatever reason, then the file is created with some default owner and group. The logrotate man page does not specify what the default is, but I'm guessing it's messagebus:adm given the large number of misc log files on my system with that mode. Unfortunately, once the file has the wrong mode, logrotate keeps patterning off it with each rotation, so you are stuck.

One solution would be to specify "nocreate" in logrotate.d/rsyslog, so just don't rely on logrotate creating empty files. it seems easier to have rsyslog or whatever create the log files with the right mode etc, instead of having rsyslog do it 99.9% of the time, but logrotate the other 0.1% of the time and you have to keep them in sync. Or if creating empty log files at rotate time is important, use the create <owner> <group> option in logrotate.d/rsyslog to specify the right mode for the log files.

[nick parlante (nick-parlante)]

Well it's a nice theory, but I can't seem to make it happen by deleting the log files and forcing a rotation.

I see that logrotate.d/rsyslog does specify the "missingok" option for these logs files. If that inhibts the "create" option for a missing log file, then my theory above theory #5 is probably out, as logrotate should not be creating these log files.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Aaron B. Russell (aaronr)]

Is this perhaps a duplicate of https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/940030 ?

I'm seeing the same thing, though I noticed it with /var/log/syslog rather than /var/log/auth.log

[Aaron B. Russell (aaronr)]

Workaround: open /etc/logrotate.d/rsyslog, and replace this line:

reload rsyslog >/dev/null 2>&1 || true

... with this line:

restart rsyslog >/dev/null 2>&1 || true

[Bryan Fullerton (fehwalker)]

Agree that this does seem to be a uplicate of 940030, marking it as such.