logrotate never recovers if the statefile is corrupted

Bug #1709670 reported by Ramon Grullon on 2017-08-09
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
logrotate (Debian)
New
Unknown
logrotate (Ubuntu)
Medium
Eric Desrochers
Trusty
Medium
Eric Desrochers
Xenial
Medium
Eric Desrochers
Zesty
Medium
Eric Desrochers
Artful
Medium
Eric Desrochers

Bug Description

[Impact]

logrotate never recovers if the statefile is corrupted unless you remove it or fix the corruption by hand.

Impact scenarios :

- System could eventually run out of disk space on a separate partition if mounted in "/var" or specifically "/var/log" or even worst if "/var/log" is on the same partition as "/" it could create even more damage if by any chance the partition is running out of free space.

- System keep updating the same files over and over, creating large size logfiles.

- ...

[Test Case]

- Install logrotate
- Run "/etc/cron.daily/logrotate" ## The first logrotate run will generate the statefile "var/lib/logrotate/status"
- Modify "/var/lib/logrotate/status" by removing the first line in order to corrupt the file
- Re-run "/etc/cron.daily/logrotate" and one will get the following error : "error: bad top line in state file /var/lib/logrotate/status" every time you run logrotate

Unless you remove the statefile and start again or fix the corruption by hand.

 * Additionally, I will run the /path_to_source/test/test script as a dogfooding that does ~72 tests.

[Regression Potential]

 * Risk of potential regression is low, and IMHO couldn't be worst than the actual situation where logrotate simply doesn't recover from a corrupt statefile.

 * The current patch does recover (after verification) and has been through some upstream CI validation, community feedbacks, et al.

 * Additionally, I will run the /path_to_source/test/test script as a dogfooding that does ~72 tests.

[Other Info]

* Upstream commit:
https://github.com/logrotate/logrotate/commit/b9d82003002c98370e4131a7e43c76afcd23306a

* Upstream bug:
https://github.com/logrotate/logrotate/issues/45

* Debian bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871592

Eric Desrochers (slashd) wrote :

Using the above reproducer ^ ....

I confirm that logrotate never recover if the statefile is corrupted unless you remove it or fix the corruption by hand.

The upstream fix introduce in 3.12.0-18 [1] is fixing the issue[2] (tested using Artful so far)

Since all supported|devel releases are lower than 3.12 [3], then all releases are affected.

[1] - https://github.com/logrotate/logrotate/commit/b9d82003002c98370e4131a7e43c76afcd23306a

# git describe --contains b9d8200
3.12.0~18

[2] - https://github.com/logrotate/logrotate/issues/45

[3] - rmadison
logrotate | 3.8.7-1ubuntu1 | trusty | source, amd64, arm64, armhf, i386, powerpc, ppc64el
logrotate | 3.8.7-1ubuntu1 | vivid | source, amd64, arm64, armhf, i386, powerpc, ppc64el
logrotate | 3.8.7-1ubuntu1.1 | trusty-updates | source, amd64, arm64, armhf, i386, powerpc, ppc64el
logrotate | 3.8.7-2ubuntu2 | xenial | source, amd64, arm64, armhf, i386, powerpc, ppc64el, s390x
logrotate | 3.8.7-2ubuntu2.16.04.1 | xenial-updates | source, amd64, arm64, armhf, i386, powerpc, ppc64el, s390x
logrotate | 3.8.7-2ubuntu3 | zesty | source, amd64, arm64, armhf, i386, ppc64el, s390x
logrotate | 3.8.7-2ubuntu3 | artful | source, amd64, arm64, armhf, i386, ppc64el, s390x

Regards,
Eric

tags: added: sts sts-sru-needed
Changed in logrotate (Ubuntu Artful):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Eric Desrochers (slashd)
Eric Desrochers (slashd) on 2017-08-09
description: updated
Eric Desrochers (slashd) on 2017-08-09
description: updated
Eric Desrochers (slashd) wrote :

Debdiff for devel release Artful.

tags: added: patch
Changed in logrotate (Ubuntu Trusty):
status: New → Confirmed
Changed in logrotate (Ubuntu Zesty):
status: New → Confirmed
Changed in logrotate (Ubuntu Xenial):
status: New → Confirmed
Eric Desrochers (slashd) on 2017-08-09
Changed in logrotate (Ubuntu Zesty):
assignee: nobody → Eric Desrochers (slashd)
Changed in logrotate (Ubuntu Xenial):
assignee: nobody → Eric Desrochers (slashd)
Changed in logrotate (Ubuntu Trusty):
assignee: nobody → Eric Desrochers (slashd)
Eric Desrochers (slashd) on 2017-08-09
description: updated
Nish Aravamudan (nacc) wrote :

Sponsored to artful.

Eric Desrochers (slashd) on 2017-08-09
description: updated
description: updated
tags: removed: patch
Eric Desrochers (slashd) on 2017-08-09
Changed in logrotate (Ubuntu Zesty):
status: Confirmed → In Progress
Changed in logrotate (Ubuntu Trusty):
status: Confirmed → In Progress
Changed in logrotate (Ubuntu Xenial):
status: Confirmed → In Progress
Ramon Grullon (rgrullon) wrote :

Tested the patch package (pre-SRU) on trusty and it works as expected.

Eric Desrochers (slashd) on 2017-08-09
Changed in logrotate (Ubuntu Artful):
status: In Progress → Fix Committed
Eric Desrochers (slashd) on 2017-08-10
description: updated
description: updated
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package logrotate - 3.8.7-2ubuntu4

---------------
logrotate (3.8.7-2ubuntu4) artful; urgency=medium

  * logrotate does not ever recover from a corrupted statefile (LP: #1709670)
    - d/p/do-not-treat-failure-of-readState-as-fatal.patch
    (Backported from commit b9d82003002c98370e4131a7e43c76afcd23306a)

 -- Eric Desrochers <email address hidden> Wed, 09 Aug 2017 11:39:36 -0700

Changed in logrotate (Ubuntu Artful):
status: Fix Committed → Fix Released
Eric Desrochers (slashd) wrote :

Debdiff for stable release : Trusty/14.04 LTS lp1709670-logrotate-trusty.debdiff

Eric Desrochers (slashd) wrote :

Debdiff for stable release : Zesty/17.04

Eric Desrochers (slashd) wrote :

Debdiff for stable release : Xenial/16.04 LTS

Changed in logrotate (Ubuntu Trusty):
importance: Undecided → Medium
Changed in logrotate (Ubuntu Xenial):
importance: Undecided → Critical
importance: Critical → Medium
Changed in logrotate (Ubuntu Zesty):
importance: Undecided → Medium
Eric Desrochers (slashd) wrote :

Trusty, Xenial & Zesty has been sponsored by myself in their respective upload queues.

Now waiting for the SRU verification team approval for the package to start building in $RELEASE-proposed for the testing phase.

- Eric

tags: added: sts-sponsor-done

Hello Ramon, or anyone else affected,

Accepted logrotate into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/logrotate/3.8.7-2ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in logrotate (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-zesty
Changed in logrotate (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Łukasz Zemczak (sil2100) wrote :

Hello Ramon, or anyone else affected,

Accepted logrotate into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/logrotate/3.8.7-2ubuntu2.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in logrotate (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed-trusty
Łukasz Zemczak (sil2100) wrote :

Hello Ramon, or anyone else affected,

Accepted logrotate into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/logrotate/3.8.7-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in logrotate (Debian):
status: Unknown → New
Eric Desrochers (slashd) wrote :

### VERIFICATION XENIAL ###

* Couldn't reproduce the above situation using the proposed package version "3.8.7-2ubuntu2.16.04.2" using Xenial/16.04 LTS.

* Ran the testsuite against "3.8.7-2ubuntu2.16.04.2" and no errors were found, apart from a recurrent one which look to me like it did succeed but reported a failure. (false positive).
(It has the exact same report result as previous version "3.8.7-2ubuntu2.16.04.1")

==
Testsuite Full Report
--
# ./test
setfacl failed on this system. ACL tests will not be executed.
./test: line 33: type: selinuxenabled: not found
SELinux disabled. SELinux tests will not be executed.
Running test 1
Running test 2
Running test 3
Running test 4
Running test 5
Running test 6
Skipping SELinux part of test 6
Running test 7
Running test 8
Running test 9
Running test 10
Skipping SELinux part of test 10
Running test 11
Running test 12
Running test 13
Running test 14
Running test 15
Running test 16
Running test 17
Running test 18
Running test 19
Running test 20
Running test 21
Running test 22
Running test 23
Running test 24
Running test 25
Running test 26
Running test 27
Running test 28
Running test 29
Running test 30
Running test 31
Skipping test 32: no ACL support
Skipping test 33: no ACL support
Running test 34
Skipping test 35: no ACL support
Running test 36
Running test 37
Running test 38
Running test 39
Running test 40
Running test 41
Running test 42
Running test 43
Running test 44
Running test 45
Running test 46
Skipping SELinux part of test 6
Running test 47
Skipping SELinux test 47
Running test 48
Skipping test 48: no ACL support
Running test 49
Running test 50
Running test 51
Running test 52
Running test 53
Running test 72
error: destination /tmp/logrotate-3.8.7/test/test.log.1.gz already exists, renaming to /tmp/logrotate-3.8.7/test/test.log.1.gz-2017081520.backup
--

- Eric

tags: added: verification-done-xenial
removed: verification-needed-xenial
Taylor Lewick (tlewick) wrote :

We tested this patched package version on ubuntu 14.04, via instructions found https://wiki.ubuntu.com/Testing/EnableProposed.

We tested by manually creating a 2.0 GB file via fallocate under the nginx log directory, and instructing the logrotate nginx config file to rotate any file > 1.0 GB.

This worked, and then I manually corrupted the logrotate status file by
1) removing the top line, creating a new test log file, and verifying logrotate ran.
2) adding a line break in a file other than the top line, creating a new test log file, and verifying logrotate ran.

Patch Behavior appears to be working as designed.

Thank you,
Taylor

Eric Desrochers (slashd) on 2017-08-16
tags: added: verification-done-trusty
removed: verification-needed-trusty

The verification of the Stable Release Update for logrotate has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package logrotate - 3.8.7-2ubuntu2.16.04.2

---------------
logrotate (3.8.7-2ubuntu2.16.04.2) xenial; urgency=medium

  * logrotate does not ever recover from a corrupted statefile (LP: #1709670)
    - d/p/do-not-treat-failure-of-readState-as-fatal.patch
    (Backported from commit b9d82003002c98370e4131a7e43c76afcd23306a)

 -- Eric Desrochers <email address hidden> Wed, 09 Aug 2017 16:02:21 -0400

Changed in logrotate (Ubuntu Xenial):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package logrotate - 3.8.7-1ubuntu1.2

---------------
logrotate (3.8.7-1ubuntu1.2) trusty; urgency=medium

  * logrotate does not ever recover from a corrupted statefile (LP: #1709670)
    - d/p/do-not-treat-failure-of-readState-as-fatal.patch
    (Backported from commit b9d82003002c98370e4131a7e43c76afcd23306a)

 -- Eric Desrochers <email address hidden> Wed, 09 Aug 2017 15:55:36 -0400

Changed in logrotate (Ubuntu Trusty):
status: Fix Committed → Fix Released
Eric Desrochers (slashd) wrote :

### VERIFICATION ZESTY ###

* Couldn't reproduce the above situation using the proposed package version "3.8.7-2ubuntu3.1" using Zesty/17.04.

* Ran the testsuite against "3.8.7-2ubuntu3.1" "/usr/sbin/logrotate" binary and no errors were found, apart from a recurrent one which look to me like it did succeed but reported a failure. (false positive).
(It has the exact same report result as previous version "3.8.7-2ubuntu3")

==
Testsuite Full Report
--
# ./test
setfacl failed on this system. ACL tests will not be executed.
./test: line 33: type: selinuxenabled: not found
SELinux disabled. SELinux tests will not be executed.
Running test 1
Running test 2
Running test 3
Running test 4
Running test 5
Running test 6
Skipping SELinux part of test 6
Running test 7
Running test 8
Running test 9
Running test 10
Skipping SELinux part of test 10
Running test 11
Running test 12
Running test 13
Running test 14
Running test 15
Running test 16
Running test 17
Running test 18
Running test 19
Running test 20
Running test 21
Running test 22
Running test 23
Running test 24
Running test 25
Running test 26
Running test 27
Running test 28
Running test 29
Running test 30
Running test 31
Skipping test 32: no ACL support
Skipping test 33: no ACL support
Running test 34
Skipping test 35: no ACL support
Running test 36
Running test 37
Running test 38
Running test 39
Running test 40
Running test 41
Running test 42
Running test 43
Running test 44
Running test 45
Running test 46
Skipping SELinux part of test 6
Running test 47
Skipping SELinux test 47
Running test 48
Skipping test 48: no ACL support
Running test 49
Running test 50
Running test 51
Running test 52
Running test 53
Running test 72
error: destination /tmp/logrotate-3.8.7/test/test.log.1.gz already exists, renaming to /var/tmp/logrotate-3.8.7/test/test.log.1.gz-2017082114.backup
--

- Eric

tags: added: verification-done-zesty
removed: verification-needed verification-needed-zesty
Eric Desrochers (slashd) on 2017-08-21
tags: added: verification-needed-zesty
removed: verification-done-zesty
tags: added: verification-done-zesty
removed: verification-needed-zesty
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package logrotate - 3.8.7-2ubuntu3.1

---------------
logrotate (3.8.7-2ubuntu3.1) zesty; urgency=medium

  * logrotate does not ever recover from a corrupted statefile (LP: #1709670)
    - d/p/do-not-treat-failure-of-readState-as-fatal.patch
    (Backported from commit b9d82003002c98370e4131a7e43c76afcd23306a)

 -- Eric Desrochers <email address hidden> Wed, 09 Aug 2017 11:25:51 -0400

Changed in logrotate (Ubuntu Zesty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.