[SRU] php5-fpm logrotate errors after package switched to upstart

[Andre Tomt]

[Impact]

 * After the switch to Upstart in Saucy, php5-fpm log rotation began to break. It was reported that individuals were getting weekly cron warning emails during logrotation.

[Test Case]

 * This test case happens with automatic rotation of logs, in which weekly cron warnings get triggered. With the fix, these weekly cron warnings for log rotation should not be triggered.

[Regression Potential]

 * This change is an upstream change that appears to have been present in the repositories in Debian and even further upstream for some time without regressions.

 * Regression potential is minimal, if only in cases where the pid gets lost, and other issues crop up.

[Other Info]

 * This bug is referred to by the MIR bug for php5-fpm (#1267255)

[Original Description]

Getting weekly cron warning emails during logrotation after php5-fpm package switched to upstart in Saucy.

root@ops:~# lsb_release -rd
Description: Ubuntu Saucy Salamander (development branch)
Release: 13.10

root@ops:~# apt-cache policy php5-fpm
php5-fpm:
  Installed: 5.5.3+dfsg-1ubuntu1
  Candidate: 5.5.3+dfsg-1ubuntu1
  Version table:
 *** 5.5.3+dfsg-1ubuntu1 0
        500 http://uk.archive.ubuntu.com/ubuntu/ saucy/universe amd64 Packages
        100 /var/lib/dpkg/status

root@ops:~# cat /etc/logrotate.d/php5-fpm
/var/log/php5-fpm.log {
 rotate 12
 weekly
 missingok
 notifempty
 compress
 delaycompress
 postrotate
  invoke-rc.d php5-fpm reopen-logs > /dev/null
 endscript
}

root@ops:~# dpkg -S /etc/logrotate.d/php5-fpm
php5-fpm: /etc/logrotate.d/php5-fpm

root@ops:~# invoke-rc.d php5-fpm reopen-logs
initctl: invalid command: reopen-logs
Try `initctl --help' for more information.
invoke-rc.d: initscript php5-fpm, action "reopen-logs" failed.

[Raoul Bhatia (raoul-bhatia)]

Is there any way to quickly address this problem by myself?

[Ondřej Surý (ondrej)]

Just change the reopen-logs in /etc/logrotate.d/php5-fpm to reload (or restart).

[Harry (harry-e)]

orion1864:~# lsb_release -rd
Description: Ubuntu Trusty Tahr (development branch)
Release: 14.04
orion1864:~# apt-cache policy php5-fpm
php5-fpm:
  Installed: 5.5.3+dfsg-1ubuntu3
  Candidate: 5.5.3+dfsg-1ubuntu3
  Version table:
 *** 5.5.3+dfsg-1ubuntu3 0
        500 http://ubuntu.intergenia.de/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status
orion1864:~# cat /etc/logrotate.d/php5-fpm
/var/log/php5-fpm.log {
 rotate 12
 weekly
 missingok
 notifempty
 compress
 delaycompress
 postrotate
  invoke-rc.d php5-fpm reopen-logs > /dev/null
 endscript
}
orion1864:~# dpkg -S /etc/logrotate.d/php5-fpm
php5-fpm: /etc/logrotate.d/php5-fpm
orion1864:~# invoke-rc.d php5-fpm reopen-logs
initctl: invalid command: reopen-logs
Try `initctl --help' for more information.
invoke-rc.d: initscript php5-fpm, action "reopen-logs" failed.

[Mike (web-g)]

Also exists in 13.10- Package info is: PHP 5.5.3-1ubuntu2.1 (fpm-fcgi) (built: Dec 12 2013 04:24:43)

[Ondřej Surý (ondrej)]

This should help:

$ git diff HEAD~2
diff --git a/debian/php5-fpm.init b/debian/php5-fpm.init
index b8b2fb3..13485ce 100644
--- a/debian/php5-fpm.init
+++ b/debian/php5-fpm.init
@@ -16,7 +16,7 @@ DESC="PHP5 FastCGI Process Manager"
 NAME=php5-fpm
 DAEMON=/usr/sbin/$NAME
 DAEMON_ARGS="--daemonize --fpm-config /etc/php5/fpm/php-fpm.conf"
-PIDFILE=/var/run/php5-fpm.pid
+PIDFILE=/run/php5-fpm.pid
 TIMEOUT=30
 SCRIPTNAME=/etc/init.d/$NAME

diff --git a/debian/php5-fpm.logrotate b/debian/php5-fpm.logrotate
index 328193e..f794c60 100644
--- a/debian/php5-fpm.logrotate
+++ b/debian/php5-fpm.logrotate
@@ -6,6 +6,6 @@
        compress
        delaycompress
        postrotate
- invoke-rc.d php5-fpm reopen-logs > /dev/null
+ kill -USR1 $(cat /run/php5-fpm.pid) > /dev/null
        endscript
 }

[Ondřej Surý (ondrej)]

And this fix will be included in 5.5.9+dfsg-2 (and when Ubuntu people merge it to their own copies of src:php5).

[Tommy Berglund (tommy-k-berglund)]

Still exists!

/etc/cron.daily/logrotate:
initctl: invalid command: reopen-logs
Try `initctl --help' for more information.
invoke-rc.d: initscript php5-fpm, action "reopen-logs" failed.
error: error running non-shared postrotate script for /var/log/php5-fpm.log of '/var/log/php5-fpm.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

Description: Ubuntu Trusty Tahr (development branch)
Release: 14.04

php5-fpm:
  Installerad: 5.5.9+dfsg-1ubuntu1
  Kandidat: 5.5.9+dfsg-1ubuntu1
  Versionstabell:
 *** 5.5.9+dfsg-1ubuntu1 0
        500 http://se.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status

[Michael Shadle (mshadle)]

will this be backported to php5-oldstable as well? I'm running saucy, which means php5-oldstable is PHP 5.5.3 currently. there is no PHP 5.4 from dotdeb. I am a little hesitant to change to just "php5" instead of "php5-oldstable" as I don't want to inherit PHP 5.6/something else in the future on accident.

[Ondřej Surý (ondrej)]

php5-5.4 from php5-oldstable doesn't have an upstart script, so this bug is not there.

[Michael Shadle (mshadle)]

yeah, woops. I did apt-add-repository ppa:ondrej/php5-oldstable, but apparently it redirects to ondrej/php5 without me realizing it. I do appreciate the new upstart style approach to the init script, that was a great change. I was concerned converting initscripts to upstart would lose custom actions and things like "reload" (which I see is commented out currently)

That's unrelated to this though.

[Holger Mauermann (mauermann)]

When will the fix be rolled out? I got some updates to PHP since ondrej proposed a fix in comment 6, but this bug still exists. There are only 3 weeks until the Trusty LTS release...

[Luca Vignaroli (voidburn)]

Running php5-fpm 5.5.9+dfsg-1ubuntu4 on a 14.04 LTS on OVH servers and this bug is there, manually switched to reload.

[Robert Pendell (shinji257)]

Trusty 14.04 is out and the current php5-fpm package still has this issue as reported in my cron job task.

/etc/cron.daily/logrotate:
initctl: invalid command: reopen-logs
Try `initctl --help' for more information.
invoke-rc.d: initscript php5-fpm, action "reopen-logs" failed.
error: error running non-shared postrotate script for /var/log/php5-fpm.log of '/var/log/php5-fpm.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

Now then my question is why not access the init.d script (apparently still included) via the services command or directly? I noted at least one other service that does the same.

For now I replaced invoke-rc.d with service and I guess I'll find out in a week or so how it goes.

[Luca Vignaroli (voidburn)]

@Robert

I can tell you now: it still loses the pid!

/etc/cron.daily/logrotate:
reload: Unknown instance:
invoke-rc.d: initscript php5-fpm, action "reload" failed.
error: error running non-shared postrotate script for /var/log/php5-fpm.log of '/var/log/php5-fpm.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

Will try Ondřej's modifications.

[Luca Vignaroli (voidburn)]

With Ondřej's changes:

/etc/cron.daily/logrotate:
logrotate_script: line 1: kill: (3205) - No such process
error: error running non-shared postrotate script for /var/log/php5-fpm.log of '/var/log/php5-fpm.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

[Vermut (vermut)]

Maybe it's possible to add special upstart service to upstart for logrotate tasks? Like

 start php5-fpm-rotate-logs

[Mikhail Emelchenkov (mikhail.guru)]

It's not OK to have broken package for so long time, please fix it finally. I receive Cron mails every week from my servers...

[Ondřej Surý (ondrej)]

@voidburn

Could you try

kill -USR2 "$(cat /var/run/php-fpm.pid)"

instead?

I am not sure where the pid file is located in Ubuntu (it would be best to read the value from /etc/php5/fpm/php-fpm.conf)...

[Ondřej Surý (ondrej)]

Or even better:

kill -USR2 "$(cat $(< /etc/php5/fpm/php-fpm.conf sed -ne 's/^pid = \(.*\)/\1/p'))"

Just make sure the "pid =" is not commented out in /etc/phpt/fpm/php-fpm.conf...

[Justin Yu (csjustinyu)]

Change /var/log/php5-fpm.log
from invoke-rc.d php5-fpm reopen-logs > /dev/null
to invoke-rc.d php5-fpm reload-configuration > /dev/null

Then add "reload-configuration)" on the top of "reopen-logs)" on /etc/init/php5-fpm
    reload-configuration)
    reopen-logs)

[Abel Cheung (abelcheung)]

1) /run or /var/run is a non-issue: Ubuntu (at least until Trusty) should have a /var/run symlink pointing to /run. So it's the same using either "/var/run/php5-fpm.pid" or "/run/php5-fpm.pid".

2) Logrotate script is not updated even on 5.5.9+dfsg-1ubuntu4.4; if the logrotate script is updated as @ondrej said in comment #5 it works for me.

3) The reload action in upstart script is also commented out on 5.5.9+dfsg-1ubuntu4.4; if it is uncommented everything is good to go.

4) Everybody, save yourself and don't hold your breath for official update. Note that this bug is still "Unassigned", which means no Ubuntu dev is paying attention.

[Michael Heuberger (michael.heuberger)]

Having this issue here too. So, been reading through your comments but what now?

Will a bugfix come out soon? If not, what's the recommended, temporary, workaround??

[Fjodor (sune-molgaard)]

A bit over a year since the original report, fix is offered in the comments.

What is keeping the responsible devs from fixing or explaining not fixing it?

[Ivan Kurnosov (zerkms)]

Guys, seriously - Importance: high, trivial fix and no attention from maintainers?

It discredits both ubuntu and the maintainers team.

[Y (x-launchpad-x)]

I agree, this should be fixed asap!

[Gregoire Pineau (lyrixx)]

We have experienced this issue too. He had to do some black magic with monit to fix this "behavior"

[Ondřej Surý (ondrej)]

Gosh, why monit blackmagic when there's a simple fix in #5 (https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1230917/comments/5)?

[Donny Kurnia (donnykurnia)]

Solution #5 works. What took maintenance team time anyway?

[Jaime Alberto Silva (jaimealbertosilva)]

This is solved on 14.10 but still on 14.04! weird.

Is there some ppa or repo where I can get a newer version of php-fpm to get this fixed on servers?

[Robie Basak (racb)]

Presumed fixed in Vivid, so marking Fix Released, but from reading the comments it sounds like this still affects 14.04 so I'll leave a bug task for Trusty open.

php5-fpm is in universe in Ubuntu, so is community maintained and not currently looked after by Canonical's server team (see also bug 1267255). If somebody prepares an SRU and is prepared drive to the fix through though, I would be happy to review and sponsor. See https://wiki.ubuntu.com/StableReleaseUpdates#Procedure which explains what to do.

[Thomas Ward (teward)]

I"ll prep an SRU and a debdiff for this fix, if only because one of my servers is impacted.

[Thomas Ward (teward)]

Debdiff for Trusty (based off of comment #5 from Ondrej)

Test build uploaded to https://launchpad.net/~teward/+archive/ubuntu/sru-builds/+packages

[Marc Deslauriers (mdeslaur)]

Nack on the debdiff.

The package in utopic adds a script called "php5-fpm-reopenlogs" which correctly parses /etc/php5/fpm/php-fpm.conf to obtain the pid file location in instead of hardcoding it to /run/php5-fpm.pid.

Pushing an SRU that hardcodes that location may break existing setups.

Please backport the changes from the utopic package instead of using a hardcoded location.

Thanks!

[Marc Deslauriers (mdeslaur)]

Unsubscribing ubuntu-sponsors for now, please re-subscribe the group once a fixed debdiff has been attached to this bug.

[Felix Geyer (debfx)]

Attached is a debdiff that backports the logrotate changes from utopic.

[Rolf Leggewie (r0lf)]

Thanks, Felix. I'll resubscribe the sponsors team now seeing that you've provided a new patch. Please target trusty-proposed instead of trusty in the changelog next time. I guess that is easy enough to fix for any sponsor on their on the fly this time around.

[Felix Geyer (debfx)]

<release> is automatically redirected to <release>-proposed nowadays so it doesn't matter which one you use.

[Marc Deslauriers (mdeslaur)]

Debdiff in comment #35 looks good, uploading for processing by the SRU team. Thanks!

[Chris J Arges (arges)]

Hello Andre, or anyone else affected,

Accepted php5 into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/php5/5.5.9+dfsg-1ubuntu4.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Felix Geyer (debfx)]

I have verified that the error message is gone with 5.5.9+dfsg-1ubuntu4.8 and the php5-fpm logs are rotated correctly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package php5 - 5.5.9+dfsg-1ubuntu4.9

---------------
php5 (5.5.9+dfsg-1ubuntu4.9) trusty-security; urgency=medium

  * SECURITY UPDATE: potential remote code execution vulnerability when
    used with the Apache 2.4 apache2handler
    - debian/patches/bug69218.patch: perform proper cleanup in
      sapi/apache2handler/sapi_apache2.c.
    - CVE number pending
  * SECURITY UPDATE: buffer overflow when parsing tar/zip/phar
    - debian/patches/bug69441.patch: check lengths in
      ext/phar/phar_internal.h.
    - CVE number pending
  * SECURITY UPDATE: heap overflow in regexp library
    - debian/patches/CVE-2015-2305.patch: check for overflow in
      ext/ereg/regex/regcomp.c.
    - CVE-2015-2305
  * SECURITY UPDATE: move_uploaded_file filename restriction bypass
    - debian/patches/CVE-2015-2348.patch: handle nulls in
      ext/standard/basic_functions.c.
    - CVE-2015-2348
  * SECURITY UPDATE: buffer overflow in unserialize when parsing Phar
    - debian/patches/CVE-2015-2783.patch: properly check lengths in
      ext/phar/phar.c, ext/phar/phar_internal.h.
    - CVE-2015-2783
  * SECURITY UPDATE: arbitrary code exection via process_nested_data
    use-after-free
    - debian/patches/CVE-2015-2787.patch: fix logic in
      ext/standard/var_unserializer.*.
    - CVE-2015-2787
 -- Marc Deslauriers <email address hidden> Fri, 17 Apr 2015 05:28:02 -0400

[anaklupa (auburn218)]

Hi,

Still getting that error.

Here's my php version:
php --version
PHP 5.5.9-1ubuntu4.9 (cli) (built: Apr 17 2015 11:44:57)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies

/etc/cron.daily/logrotate:
error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/error.log '
initctl: invalid command: reopen-logs
Try `initctl --help' for more information.
invoke-rc.d: initscript php5-fpm, action "reopen-logs" failed.
error: error running non-shared postrotate script for /var/log/fpm-php.www.log of '/var/log/fpm-php.www.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

[Matt LaPlante (cybrmatt)]

Check that your logrotate config was actually updated?

 cd /etc/logrotate.d; mv php5-fpm.dpkg-dist php5-fpm