logrotate incorrectly rotates /dev/null to /dev/null.1

Bug #387189 reported by H.-Dirk Schmitt on 2009-06-15
52
This bug affects 6 people
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
Undecided
Unassigned
Hardy
Undecided
Unassigned
Karmic
Undecided
Unassigned
Lucid
Undecided
Unassigned
logrotate (Ubuntu)
High
Canonical Server Team
Hardy
High
Unassigned
Karmic
High
Unassigned
Lucid
High
Unassigned

Bug Description

IMPACT:
If logrotate is unable to find the logs it is expecting, it will incorrectly rotate /dev/null -> /dev/null. This can cause the session to become totally locked. Login is not possible (see bug #270781).

ADDRESSED:
A patch cherry picked from >Maverick packages resolves the config parsing when the wildcard produces an empty set (which defaulted to /dev/null).

PATCH:
See Merge proposal

TEST CASE:
Install apache2-commons (which includes a logrotate config), but have no apache running -> therefore no logs to rotate. Wait, or induce a log rotation and witness /dev/null be rotated to /dev/null.1.

REGRESSION POTENTIAL:
This should be considered low, the attached resolution has been in Debian since March 2010 and is in the Maverick release.

Related branches

tags: added: jaunty

Second desktop shows the problem (also 9.04/amd-64) after being hibernated 4 or 5 times.

Sabin Iacob (iacobs) wrote :

same here on a laptop that I rarely reboot (I just suspend to ram); after a while, /dev/null is replaced with a blank file, and /dev/null.1 appears; if /dev/null.1 exists, /dev/null.2 is created and so on

Changed in ubuntu:
status: New → Confirmed
Kamus (kamus) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage. I have classified this bug as a bug in udev and please do not confirm your own reports ;)

affects: ubuntu → udev (Ubuntu)
Changed in udev (Ubuntu):
status: Confirmed → New

I'm marking the udev task as invalid, because this is not a udev bug.

Something else on the system is overwriting /dev/null (removing the device node udev creates), and obviously is also creating those .1, .2, etc. files

This will be quite difficult to debug, you'll need to first establish whether /dev/null exists before the system is booted. Try booting with init=/bin/bash on the kernel command line and check "stat /dev/null" - is it a file there or a device node?

Changed in udev (Ubuntu):
status: New → Invalid
affects: udev (Ubuntu) → ubuntu
Changed in ubuntu:
status: Invalid → Triaged
status: Triaged → Incomplete

The problem is not depending on a reboot.
After some time the /dev/null device is moved to /dev/null.1 and a new **file** is created for /dev/null.
This triggers https://bugs.launchpad.net/ubuntu/+source/pam/+bug/270781, if the screensaver locks the screen.

Some "older" processes still using the moved /dev/null.1
Some "newer" processes writing to the ordinary file /dev/null and filling up the limited disk space on the root partition.

The problem occures here on a amd64/jaunty desktop system and a i386/jaunty netbook.
Sabin Iacob reported the same problem .

The process/package moving the file isn't determined in the moment.

Changed in ubuntu:
status: Incomplete → New

Problem occurred again.
The output of fuser -av /dev/null* is attached.

zK (zk-ngi) wrote :

Happens also to me every 2-3 days.

Upgrading to Karmic 9.10 didn't resolve the issue .

zK (zk-ngi) wrote :

I think it might be related to syslog-ng, i removed it 5 days ago and still no sign of the bug

Can't confirm.
I'm not using syslog-ng.
The problem occurs also under karmic with rsyslogd on a netbook.

The problem still occurs 2 machines
fuser -av /dev/null
shows always atop in the process list

atop isn't the reason
I have currently 5 installations (2 fresh karmic installations) with this problem.

solarw (solarw-mail) wrote :

same bug

/dev/null - is ordinal file
/dev/null.1 - is original /dev/null device file

same problem to unlock gnome-screensaver without /dev/null fix

Same bug under Lucid.

I had started to audit /dev/null and recognized that logrotate simply rotated it.

The output of ausearch -i -f /dev/null is included. It was a character device and became a regular file. I've checked every logrotate rules, but I can't guess why it happend. I will continue debugging with tracing logrotate.

Changed in ubuntu:
status: New → Incomplete
status: Incomplete → New
Fabio Marconi (fabiomarconi) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage . I have classified this bug as a bug in PKGNAME.

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://help.ubuntu.com/community/ReportingBugs.

Fabio Marconi (fabiomarconi) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please answer these questions:

* Is this reproducible?
* If so, what specific steps should we take to recreate this bug?

This will help us to find and resolve the problem.

Changed in ubuntu:
status: New → Incomplete
pberndt (phillip-berndt) wrote :

I'm pretty sure this bug is related to logrotate. When Mészáros Balázs suggested that it might be, I disabled logrotate on both my systems and did not have any problems ever since.

Fabio Marconi (fabiomarconi) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage.

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://help.ubuntu.com/community/ReportingBugs.

pberndt (phillip-berndt) on 2010-08-07
affects: ubuntu → logrotate (Ubuntu)
Fabio Marconi (fabiomarconi) wrote :

Hello pberndt
Can you please run in a terminal:
apport-collect -p logrotate 387189

Fabio Marconi (fabiomarconi) wrote :

when the reported files are attached you can mark as confirmed.

Architecture: i386
DistroRelease: Ubuntu 10.04
NonfreeKernelModules: nvidia
Package: logrotate 3.7.8-4ubuntu2
PackageArchitecture: i386
ProcVersionSignature: Ubuntu 2.6.32-24.39-generic 2.6.32.15+drm33.5
Tags: lucid
Uname: Linux 2.6.32-24-generic i686
UserGroups: adm admin audio cdrom dialout fuse lp lpadmin netdev plugdev sambashare tape vboxusers video

tags: added: apport-collected

apport information

apport information

Changed in logrotate (Ubuntu):
status: Incomplete → Confirmed

I think I tracked this down to a syntax thinko in /etc/logrotate.d/apache2 (installed by apache2.2-common), when /etc/apache2/envvars doesn't exist - which can happen easily, e.g., when you do not really need apache2 but only something from the -common package:

logrotate -d /etc/logrotate.conf 2>&1 gives me:

[...]
reading config file apache2
error: error accessing /var/log/apache2: No such file or directory
error: apache2:1 glob failed for /var/log/apache2/*.log
error: found error in /var/log/apache2/*.log , skipping
removing last 1 log configs
error: apache2:11 lines must begin with a keyword or a filename (possibly in double quotes)
error: apache2:12 missing end of line
reading config info for /etc/init.d/apache2 reload > /dev/null
                fi
        endscript
}
[...]
 weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /etc/init.d/apache2
  log does not need rotating
considering log reload
error: stat of reload failed: No such file or directory
considering log >
error: stat of > failed: No such file or directory
considering log /dev/null
  log does not need rotating
considering log fi
error: stat of fi failed: No such file or directory
considering log endscript
error: stat of endscript failed: No such file or directory
considering log }
error: stat of } failed: No such file or directory

Obviously logrotate fails to parse the output of the backticked command. And trys to interpret the following lines as filenames that should be logrotated. The only file that exists is /dev/null.

Removing the unneeded apache2 logrotate script corrects the logrotate -d output and the behaviour.

I think there could be more than one way to fix this bug:

1. in apache-common: correct the if-statement, taking into account /etc/apache2/envvars might not even exist.
2. in logrotate: make logrotate check if it is told to operate on a regular file (like savelog does)
3. let logrotate discard an erroneous config file, instead of trying to interpret too much.
...

But the quick workaround: remove /etc/logrotate.d/apache2 if you don't need it.

Andi Hechtbauer (anti-dotu) wrote :

The problem is within /etc/logrotate.d/apache2 - when /etc/apache2/envvars does not exist:

# logrotate -d /etc/logrotate.conf:

[...]
error: apache2:11 lines must begin with a keyword or a filename (possibly in double quotes)
error: apache2:12 missing end of line
reading config info for /etc/init.d/apache2 reload > /dev/null
                fi
        endscript
}
[...]
considering log reload
error: stat of reload failed: No such file or directory
considering log >
error: stat of > failed: No such file or directory
considering log /dev/null
  log does not need rotating
considering log fi
error: stat of fi failed: No such file or directory
considering log endscript
error: stat of endscript failed: No such file or directory
considering log }
error: stat of } failed: No such file or directory
[...]

Workaround: remove /etc/logrotate.d/apache2 when not needed.
Possible Fix in apache2-common: additionally test if [ -f /etc/apache2/envvars ] before using that script in backticks.
Possible Fix in logrotate: Do not rotate non-regular files, Maybe do not try to interpret everything as a filename.

Paul Tagliamonte (paultag) wrote :

Marking High status. This breaks software that is unrelated to it's self. libc will fail if this is not a character block device, and even can deny remote login ( such as on servers, in some cases ). This is an important issue to address.

Changed in logrotate (Ubuntu):
importance: Undecided → High
Dave Gilbert (ubuntu-treblig) wrote :

Based on comments #24 and #25 it looks like this is an Apache issue, so I've added apache2 - apache2 maintainers please have a look because it's got quite an impact.

Dave

Jonathan Marsden (jmarsden) wrote :

It seems to me a patch to /etc/logrotate.d/apache2 is the simplest solution.

The attached diff tries to solve the issue and also be more readable,
breaking the one complex line involving backticks into three shorter
simpler lines, no backticks needed.

There are a lot of cases to test this with, and I doubt I have yet tested all of them...
with and without a full apache2 installation, with and without apache running,
with and without making changes to the envvars file to put the PID file elsewhere...

Anyway, here is a patch that looks sane to me. Comments welcomed, testing
even more welcomed :)

tags: added: patch
Stefan Fritsch (sf-sfritsch) wrote :

This is a logrotate issue and happens if the specified logfile directory (in this case /var/log/apache2) does not exist and the postrotate script contains a closing '}'. Therefore I don't think Jonathan's patch would fix the issue completely.

The full info is at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571033 and https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/392532

The patch is included in logrotate 3.7.8-5 and newer.

Jonathan Marsden (jmarsden) wrote :

OK. Do we have a test case for which my patch fails?

If the remaining issue is that my patch contains a closing '}', that is easily "solved", because the {} around the variable name are only there for style reasons, and are not required.

Here is a modified patch without the {} . Is there a test case for which this one fails?

If this issue is deemed significant enough to SRU for Hardy and Lucid, then it seems easier to me to get a small patch to one script accepted, than to get a newer version of logrotate accepted.

Is it better to backport the logrotate patch? To do both the logrotate patch *and* fix the apache2 logrotate script?

Changed in logrotate (Ubuntu):
assignee: nobody → Canonical Server Team (canonical-server)
Dave Walker (davewalker) on 2011-02-01
summary: - /dev/null corrupted (/dev/null.1)
+ logrotate incorrectly rotates /dev/null to /dev/null.1
Dave Walker (davewalker) on 2011-02-01
description: updated
Changed in logrotate (Ubuntu Hardy):
importance: Undecided → High
status: New → Confirmed
Changed in logrotate (Ubuntu Karmic):
importance: Undecided → High
status: New → Confirmed
Changed in logrotate (Ubuntu Lucid):
importance: Undecided → High
status: New → Confirmed
Martin Pitt (pitti) wrote :

From "patch cherry picked from >Maverick" I assume that this is fixed in natty. Please set the task states appropriately.

Changed in logrotate (Ubuntu):
status: Confirmed → Fix Released
Changed in logrotate (Ubuntu Lucid):
status: Confirmed → Fix Committed
tags: added: verification-needed

Accepted logrotate into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in apache2 (Ubuntu):
status: New → Invalid
Changed in apache2 (Ubuntu Karmic):
status: New → Invalid
Changed in apache2 (Ubuntu Hardy):
status: New → Invalid
Changed in apache2 (Ubuntu Lucid):
status: New → Invalid
Dave Walker (davewalker) wrote :

@Martin, at the time this upload was worked on the tasks were not yet accepted. Thank you for updating them.

C de-Avillez (hggdh2) wrote :

Confirmed fix in logrotate 4.7.8-4ubuntu2.1 (Lucid proposed). Marking verification-done.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package logrotate - 3.7.8-4ubuntu2.1

---------------
logrotate (3.7.8-4ubuntu2.1) lucid-proposed; urgency=low

  * debian/patches/parser571033.patch: Fix the config parser to not get
    confused when a wildcard produces no results, which caused /dev/null
    to be rotated. (LP: #387189)
 -- Dave Walker (Daviey) <email address hidden> Tue, 01 Feb 2011 13:51:08 +0000

Changed in logrotate (Ubuntu Lucid):
status: Fix Committed → Fix Released
Rolf Leggewie (r0lf) on 2011-10-19
Changed in logrotate (Ubuntu Karmic):
status: Confirmed → Won't Fix
Rolf Leggewie (r0lf) wrote :

Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix".

Changed in logrotate (Ubuntu Hardy):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.