Activity log for bug #1630516

Date Who What changed Old value New value Message
2016-10-05 10:28:16 Thomas A. F. Thorne bug added bug
2016-10-05 10:29:22 Thomas A. F. Thorne bug watch added http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734688
2016-10-05 10:29:22 Thomas A. F. Thorne bug task added logrotate (Debian)
2016-10-05 11:36:53 Bug Watch Updater logrotate (Debian): status Unknown New
2016-12-08 19:03:17 Launchpad Janitor logrotate (Ubuntu): status New Confirmed
2016-12-27 00:34:31 Bug Watch Updater logrotate (Debian): status New Confirmed
2017-01-04 20:12:18 Bug Watch Updater logrotate (Debian): status Confirmed Fix Released
2017-01-26 22:30:23 rpr nospam bug added subscriber rpr nospam
2017-03-15 08:25:17 Thomas A. F. Thorne bug added subscriber Papercuts Ninjas
2017-03-15 08:56:51 Christian Ehrhardt  bug added subscriber Ubuntu Server Team
2017-03-15 08:57:04 Christian Ehrhardt  nominated for series Ubuntu Trusty
2017-03-15 08:57:04 Christian Ehrhardt  nominated for series Ubuntu Xenial
2017-03-15 08:57:04 Christian Ehrhardt  nominated for series Ubuntu Zesty
2017-03-15 08:57:04 Christian Ehrhardt  nominated for series Ubuntu Yakkety
2017-03-15 08:57:37 Christian Ehrhardt  bug added subscriber ChristianEhrhardt
2017-03-15 18:03:01 Alberto Salvia Novella summary Logrotate fails repeatedly with /etc/cron.daily/logrotate: error: error creating output file /var/log/... File exists run-parts: /etc/cron.daily/logrotate exited with return code 1 Logrotate fails to clean old system logs, allowing them to fill the hole disk
2017-03-15 18:03:31 Alberto Salvia Novella tags amd64 apport-bug trusty amd64 apport-bug trusty xenial
2017-03-15 18:03:48 Alberto Salvia Novella tags amd64 apport-bug trusty xenial amd64 apport-bug trusty xenial yakkety
2017-03-15 18:04:04 Alberto Salvia Novella tags amd64 apport-bug trusty xenial yakkety amd64 apport-bug trusty xenial yakkety zesty
2017-03-15 18:04:20 Alberto Salvia Novella bug task added hundredpapercuts
2017-03-15 18:04:30 Alberto Salvia Novella hundredpapercuts: status New Confirmed
2017-03-15 18:04:54 Alberto Salvia Novella summary Logrotate fails to clean old system logs, allowing them to fill the hole disk Logrotate doesn't clean old system logs, allowing them to fill the hole disk
2017-03-15 18:05:00 Alberto Salvia Novella hundredpapercuts: importance Undecided Critical
2017-03-15 18:05:03 Alberto Salvia Novella logrotate (Ubuntu): importance Undecided Critical
2017-03-15 18:06:32 Alberto Salvia Novella logrotate (Ubuntu): status Confirmed Triaged
2017-03-15 18:06:33 Alberto Salvia Novella hundredpapercuts: status Confirmed Triaged
2017-03-15 18:06:52 Alberto Salvia Novella summary Logrotate doesn't clean old system logs, allowing them to fill the hole disk Logrotate doesn't clean old system logs, allowing them to fill the disk
2017-03-15 18:07:29 Alberto Salvia Novella removed subscriber Papercuts Ninjas
2017-03-17 23:13:26 Brian Murray bug task added logrotate (Ubuntu Zesty)
2017-03-22 15:52:10 Christian Ehrhardt  description Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron <root@warden> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root 56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root 24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron <root@trac> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r----- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r----- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r----- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r----- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r----- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r----- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r----- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r----- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r----- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0660 root utmp rotate 1 } # system-specific logs may be configured here $ cat /etc/logrotate.d/rsyslog /var/log/syslog { rotate 7 daily missingok notifempty delaycompress compress postrotate reload rsyslog >/dev/null 2>&1 || true endscript } /var/log/mail.info /var/log/mail.warn /var/log/mail.err /var/log/mail.log /var/log/daemon.log /var/log/kern.log /var/log/auth.log /var/log/user.log /var/log/lpr.log /var/log/cron.log /var/log/debug /var/log/messages { rotate 4 weekly missingok notifempty compress delaycompress sharedscripts postrotate reload rsyslog >/dev/null 2>&1 || true endscript } manager@trac:/var/log$ cat /etc/logrotate.d/munin-node /var/log/munin/munin-node.log { daily missingok rotate 7 compress copytruncate notifempty create 640 root root } I am confident that is all the files which are relative to the configuration: manager@trac:/var/log$ rgrep syslog /etc/logrotate.* /etc/logrotate.conf:# use the syslog group by default, since this is the owning group /etc/logrotate.conf:# of /var/log/syslog. /etc/logrotate.conf:su root syslog /etc/logrotate.d/ufw: invoke-rc.d rsyslog reload >/dev/null 2>&1 || true /etc/logrotate.d/rsyslog:/var/log/syslog /etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true /etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true manager@trac:/var/log$ rgrep munin /etc/logrotate.* /etc/logrotate.d/munin-node:/var/log/munin/munin-node.log If there is anything else I can do to help diagnose the issue please let me know. Usual boiler plate bits: "trac" machine: $ lsb_release -rd Description: Ubuntu 14.04.5 LTS Release: 14.04 $ apt-cache policy logrotate logrotate: Installed: 3.8.7-1ubuntu1 Candidate: 3.8.7-1ubuntu1 Version table: *** 3.8.7-1ubuntu1 0 500 http://gb.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages 100 /var/lib/dpkg/status "warden" machine: $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ apt-cache policy logrotate logrotate: Installed: 3.8.7-2ubuntu2 Candidate: 3.8.7-2ubuntu2 Version table: *** 3.8.7-2ubuntu2 500 500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages 100 /var/lib/dpkg/status N: Ignoring file '50unattended-upgrades.ucf-dist' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: logrotate 3.8.7-1ubuntu1 ProcVersionSignature: Ubuntu 4.2.0-42.49~14.04.1-generic 4.2.8-ckt12 Uname: Linux 4.2.0-42-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.21 Architecture: amd64 Date: Wed Oct 5 11:14:53 2016 InstallationDate: Installed on 2015-12-14 (295 days ago) InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805) SourcePackage: logrotate UpgradeStatus: No upgrade log present (probably fresh install) [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron <root@warden> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root 56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root 24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron <root@trac> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r----- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r----- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r----- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r----- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r----- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r----- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r----- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r----- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r----- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp {     missingok     monthly     create 0664 root utmp     rotate 1 } /var/log/btmp {     missingok     monthly     create 0660 root utmp     rotate 1 } # system-specific logs may be configured here $ cat /etc/logrotate.d/rsyslog /var/log/syslog {             rotate 7             daily             missingok             notifempty             delaycompress             compress             postrotate                         reload rsyslog >/dev/null 2>&1 || true             endscript } /var/log/mail.info /var/log/mail.warn /var/log/mail.err /var/log/mail.log /var/log/daemon.log /var/log/kern.log /var/log/auth.log /var/log/user.log /var/log/lpr.log /var/log/cron.log /var/log/debug /var/log/messages {             rotate 4             weekly             missingok             notifempty             compress             delaycompress             sharedscripts             postrotate                         reload rsyslog >/dev/null 2>&1 || true             endscript } manager@trac:/var/log$ cat /etc/logrotate.d/munin-node /var/log/munin/munin-node.log {             daily             missingok             rotate 7             compress             copytruncate             notifempty             create 640 root root } I am confident that is all the files which are relative to the configuration: manager@trac:/var/log$ rgrep syslog /etc/logrotate.* /etc/logrotate.conf:# use the syslog group by default, since this is the owning group /etc/logrotate.conf:# of /var/log/syslog. /etc/logrotate.conf:su root syslog /etc/logrotate.d/ufw: invoke-rc.d rsyslog reload >/dev/null 2>&1 || true /etc/logrotate.d/rsyslog:/var/log/syslog /etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true /etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true manager@trac:/var/log$ rgrep munin /etc/logrotate.* /etc/logrotate.d/munin-node:/var/log/munin/munin-node.log If there is anything else I can do to help diagnose the issue please let me know. Usual boiler plate bits: "trac" machine: $ lsb_release -rd Description: Ubuntu 14.04.5 LTS Release: 14.04 $ apt-cache policy logrotate logrotate:   Installed: 3.8.7-1ubuntu1   Candidate: 3.8.7-1ubuntu1   Version table:  *** 3.8.7-1ubuntu1 0         500 http://gb.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages         100 /var/lib/dpkg/status "warden" machine: $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ apt-cache policy logrotate logrotate:   Installed: 3.8.7-2ubuntu2   Candidate: 3.8.7-2ubuntu2   Version table:  *** 3.8.7-2ubuntu2 500         500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages         100 /var/lib/dpkg/status N: Ignoring file '50unattended-upgrades.ucf-dist' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: logrotate 3.8.7-1ubuntu1 ProcVersionSignature: Ubuntu 4.2.0-42.49~14.04.1-generic 4.2.8-ckt12 Uname: Linux 4.2.0-42-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.21 Architecture: amd64 Date: Wed Oct 5 11:14:53 2016 InstallationDate: Installed on 2015-12-14 (295 days ago) InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805) SourcePackage: logrotate UpgradeStatus: No upgrade log present (probably fresh install)
2017-03-27 18:13:33 Michael Herold bug added subscriber Michael Herold
2017-04-05 07:47:32 Christian Ehrhardt  bug added subscriber Ubuntu Sponsors Team
2017-04-05 09:15:20 Robie Basak bug task added logrotate (Ubuntu Trusty)
2017-04-05 09:15:33 Robie Basak bug task added logrotate (Ubuntu Xenial)
2017-04-05 09:15:46 Robie Basak bug task added logrotate (Ubuntu Yakkety)
2017-04-05 09:19:22 Robie Basak logrotate (Ubuntu Trusty): status New In Progress
2017-04-05 09:19:25 Robie Basak logrotate (Ubuntu Xenial): status New In Progress
2017-04-05 09:19:30 Robie Basak logrotate (Ubuntu Yakkety): status New In Progress
2017-04-05 09:19:48 Robie Basak logrotate (Ubuntu Zesty): status Triaged Fix Committed
2017-04-05 12:35:09 Launchpad Janitor logrotate (Ubuntu Zesty): status Fix Committed Fix Released
2017-04-05 13:02:10 Christian Ehrhardt  logrotate (Ubuntu Trusty): importance Undecided High
2017-04-05 13:02:12 Christian Ehrhardt  logrotate (Ubuntu Xenial): importance Undecided High
2017-04-05 13:02:17 Christian Ehrhardt  logrotate (Ubuntu Yakkety): importance Undecided High
2017-04-14 06:42:37 Steve Langasek logrotate (Ubuntu Yakkety): status In Progress Fix Committed
2017-04-14 06:42:40 Steve Langasek bug added subscriber Ubuntu Stable Release Updates Team
2017-04-14 06:42:46 Steve Langasek bug added subscriber SRU Verification
2017-04-14 06:42:53 Steve Langasek tags amd64 apport-bug trusty xenial yakkety zesty amd64 apport-bug trusty verification-needed xenial yakkety zesty
2017-04-14 06:43:19 Steve Langasek logrotate (Ubuntu Xenial): status In Progress Fix Committed
2017-04-14 06:43:23 Steve Langasek removed subscriber Ubuntu Sponsors Team
2017-04-14 06:44:15 Steve Langasek logrotate (Ubuntu Trusty): status In Progress Fix Committed
2017-04-14 10:33:39 paz bug added subscriber paz
2017-04-26 15:27:38 Thomas A. F. Thorne tags amd64 apport-bug trusty verification-needed xenial yakkety zesty amd64 apport-bug trusty verification-done-xenial verification-needed xenial yakkety zesty
2017-05-10 15:50:09 Launchpad Janitor logrotate (Ubuntu Xenial): status Fix Committed Fix Released
2017-06-14 14:10:08 Chris J Arges logrotate (Ubuntu Trusty): importance High Medium
2017-06-14 14:10:14 Chris J Arges logrotate (Ubuntu Trusty): importance Medium High
2017-06-14 14:13:23 Chris J Arges tags amd64 apport-bug trusty verification-done-xenial verification-needed xenial yakkety zesty amd64 apport-bug trusty verification-done-trusty verification-done-xenial verification-needed xenial yakkety zesty
2017-06-14 14:35:46 Chris J Arges tags amd64 apport-bug trusty verification-done-trusty verification-done-xenial verification-needed xenial yakkety zesty amd64 apport-bug trusty verification-done-trusty verification-done-xenial verification-done-yakkety verification-needed xenial yakkety zesty
2017-06-14 14:35:55 Chris J Arges tags amd64 apport-bug trusty verification-done-trusty verification-done-xenial verification-done-yakkety verification-needed xenial yakkety zesty amd64 apport-bug trusty verification-done-trusty verification-done-xenial verification-done-yakkety xenial yakkety zesty
2017-06-14 21:23:31 Launchpad Janitor logrotate (Ubuntu Trusty): status Fix Committed Fix Released
2017-06-14 21:23:36 Chris J Arges removed subscriber Ubuntu Stable Release Updates Team
2017-06-14 21:23:52 Launchpad Janitor logrotate (Ubuntu Yakkety): status Fix Committed Fix Released
2017-08-09 10:02:01 Joshua Bayfield hundredpapercuts: status Triaged Fix Released