"reload php5-fpm" crashes

[Andre Tomt]

[Impact]

On releases running upstart (those older than Vivid), the php5-fpm service crashes on reload.

[Workaround]

Create /etc/init/php5-fpm.override and add the line "reload signal SIGUSR2" to it.

[Original Description]

"reload php5-fpm" with upstart does not work.

php5-fpm will crash and leave non-working child processes hanging around that will block any recovery until they are killed off manually.

This is because php5-fpm does not handle SIGHUP. It results in an uncontrolled exit. After telling upstart to instead use SIGUSR2 on reload*, things work as expected.

* "reload signal SIGUSR2" in upstart php5-fpm.conf

Signals handled by php5-fpm seems to be:
* SIGINT, SIGTERM: immediate termination
* SIGQUIT: graceful stop
* SIGUSR1: re-open log file
* SIGUSR2: graceful reload of all workers + reload of fpm conf/binary

Ubuntu 13.10, amd64
ii php5-fpm 5.5.3+dfsg-1ubuntu2 amd64 server-side, HTML-embedded scripting language (FPM-CGI binary)

[Andre Tomt (andre-tomt)]

Note that the move to upstart for php5-fpm happened with Ubuntu 13.10
Its not relevant for older releases.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Confirmed: php5-fpm(8) documents USR2 as the reload signal, and the upstart script defaults to HUP.

Workaround (untested): add the line "reload signal USR2" to /etc/init/php5-fpm.conf.

This also exists in Debian so should be filed there too. I'll do it when I look at fixing this in Ubuntu, unless someone else gets to it first.

[Ondřej Surý (ondrej)]

As far as I remember I have discussed this on the debian-devel and the reload signal was added to upstart just recently.

Robie, could you please confirm that the older versions of upstart won't cry too much if they encounter unknown directive?

[Robie Basak (racb)]

@Ondřej

I just tested upstart 1.5-0ubuntu7.2 from Precise. This fails with "signal USR2" with the error "Unknown stanza", and refuses to process the job. So this will only work with upstart >= 1.10.0. It looks like this first became available in Debian unstable only a few days ago.

Does this block you?

[Robie Basak (racb)]

Oh, I've just seen your fix uploaded in 5.5.6+dfsg-1, sorry.

[Peter Wang (peterwang)]

Hi,

Great to see that the bug was fixed. However, what could be the solution for old version (< 1.10.0) of upstart? For instance, in ubuntu precise, the version of upstart is "1.5-0ubuntu7.2", is there any suggested workaround for such case? Thanks!

[Peter Wang (peterwang)]

I have found that I can still use "/etc/init.d/php5-fpm" to do "reload", however, this script doesn't work well with php5-fpm processes that are started by "service php5-fpm start", for instance, "/etc/init.d/php5-fpm stop" sometimes can't stop all the processes, I have not looked into the cause of that yet. So it's better not mix these 2 ways.

[Giacomo Orlandi (giaco-orlandi)]

Hi,

We are using Precise LTS too here.
The init script from the php-fpm package in Ondřej's php5-oldstable PPA (PHP 5.4) works on Precise.
That's using start-stop-daemon instead of upstart.

So a fix for Precise, and possibly other older distributions, is to use that init script also on PHP 5.5.
This is the command that is run there:
start-stop-daemon --stop --signal USR2 --quiet --pidfile /var/run/php5-fpm.pid --name php5-fpm

Peter Wang: I can't see any difference between using "service" or the "init.d" script here.
I experience the same behaviour I use one, or the other or a mix of the 2.

[Peter Wang (peterwang)]

Hi,

Giacomo Orlandi: sorry that I didn't make it clear, by "/etc/init.d/php5-fpm" I meant what you just mentioned (The init script from the php-fpm package in Ondřej's php5-oldstable PPA), not the symlink provided by the php5-fpm package.

[Richard (richard-ibbotson)]

Did an upgrade on my Ubuntu 14.04 server on the 7th of March. php5-fpm died immediately. Adding lines of code such as "reload signal SIGUSR2" or taking them away makes no difference. Running php5-fpm -v produces this print out "Segmentation fault" Result of php5 -v is................

 PHP 5.5.9-1ubuntu2 (cli) (built: Mar 3 2014 21:10:03)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with XCache v3.1.0, Copyright (c) 2005-2013, by mOo
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
    with XCache Optimizer v3.1.0, Copyright (c) 2005-2013, by mOo
    with XCache Cacher v3.1.0, Copyright (c) 2005-2013, by mOo
    with XCache Coverager v3.1.0, Copyright (c) 2005-2013, by mOo

[mithras (jsluiman)]

I'm on precise64 with ondrej's PPA for php 5.5 and experiencing the same issues. Using "service" results in the bug report's error. Using init.d gives me:

stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.15" (uid=1001 pid=21912 comm="stop php5-fpm ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")

Enabling the reload in /etc/init/php-fpm gives me this response on service php5-fpm stop:

stop: Method "Get" with signature "ss" on interface "org.freedesktop.DBus.Properties" doesn't exist

Now I am killing manually the php processes and with service I am spinning them up again (all tested inside vagrant), but I'd like to know what's the solution for 12.04 ppl now using php 5.5.

[Renan Gonçalves (renan-saddam)]

The package 5.5.9+dfsg-1ubuntu3 have the correct upstart script with "reload signal USR2", which works great on Ubuntu Trusty since it uses upstart >= 1.10.0

For fixing the https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1272788 bug a new package was released, called "5.5.9+dfsg-1ubuntu4", which comments out "reload signal USR2".
Making it work on Precise but breaking on Trusty.

Installing a new Ubuntu machine with the latest version and latest PHP should be the priority to work out of the box. No adjustments should be needed.
If anyone wants to use Precise with the latest PHP version, adjustments to the upstart script are okay.

Hence, IMO this fix should be reversed.

[Laurent Declercq (l-declercq)]

I agreed totally with Renan... PHP5-FPM reload is broken meaning that restart is mandatory ATM (which is not a way to go for shared hosting environment where configuration should be reloaded without having to do a full restart)...

[Tom Fotherby (a-launchpad-tomfotherby-com)]

We ran into this bug while upgrading our servers from 12.04 LTS to 14.04 LTS. It is serious for us. When we deploy our app, we need to do `service php5-fpm reload` so that the php opcode cache is flushed, otherwise old files will still be served. Doing a `php5-fpm restart` isn't a option as it does not handle existing connections gracefully.

We would really really appreciate for this bug to be fixed in the Ubuntu repos. It is really not a good bug to have in a LTS release for folks who run PHP. It will catch many people out.

[Panagiotis Moustafellos (pmoust)]

This should be treated ASAP, atm Ubuntu 14.04 LTS php5-fpm package cannot gracefully reload workers with new configuration.
Restart is not an option in a production environment.

[Robie Basak (racb)]

Some possible workarounds (off the top of my head):

1) Restart instead.
2) Send a USR2 signal to the correct PID manually (get it with "status php5-fpm").
3) Create /etc/init/php5-fpm.override file with the single line "reload signal USR2" in it.

Please report which of these workarounds works for you, and I'll change the bug description at the top so that others will see it easily.

[Michael Shadle (mshadle)]

+1

We have this issue too. Seems the "reload signal USR2" being in the official php5-fpm.conf is still commented out. Not sure why, as it should be able to be pinned just to recent versions of Upstart / releases.

I have used Robie's suggestion #3 above in an .override file, and that seems to work at the moment.

option #1 isn't really an option (same reasons in comment #15). option #2 is the best bet. Just a little more robustness is required in the Upstart script. Appreciate all the work so far, Upstart wasn't exactly designed in the beginning to support all these other actions.

[Ondřej Surý (ondrej)]

I think it's not exactly clear here, why this cannot be fixed right away in Ubuntu 14.04 LTS.

The main problem is that 'upstart' is not following Postel's principle (be benevolent in what you receive, and strict in what you send) and barfs and breaks whole release upgrade process if it encounters directive it doesn't know about.

In this case the problem gets triggered on upgrades from Ubuntu 12.04 when upstart has been upgraded to 1.12.1, but the PID 1 (/sbin/init) is still binary from upstart 1.5 that fails to start/stop php5-fpm and thus breaking the whole upgrade process.

This has by no means an easy fix and only possible solution would be to fix that first in precise upstart to just ignore (print warning) on unknown directives in init files and then fix this in Ubuntu 14.04.

Hmm, or we can just patch php5-fpm to reload on SIGHUP :).

[Ondřej Surý (ondrej)]

@Robie: Try pulling fbdc116d6023141060d515949f38fcaa61ef2ed7 from master-5.5 branch.

I have implemented:

> Hmm, or we can just patch php5-fpm to reload on SIGHUP

That would just make everyone happy without fixing upstart.

[Ondřej Surý (ondrej)]

JFTR the ppa:ondrej/php5-oldstable, ppa:ondrej/php5 and ppa:php5-5.6 has been updated (or will be updated shortly) with patched PHP 5.4, 5.5 and 5.6 that accept SIGHUP as reload signal and thus it doesn't break with upstart.

[Panagiotis Moustafellos (pmoust)]

@ondrej , I've used your ppa that provides PHP 5.5.14-2+deb.sury.org~trusty+1 (cli) (built: Jul 2 2014 12:14:44) , doing service php5-fpm reload or manually sending kill -1 to master fpm process does not seem to alter the master PID.
Shouldn't the master process respawn itself as well?
I was expecting to see another PID for the master fpm process.

[Panagiotis Moustafellos (pmoust)]

On 12.04
pmoust@webserver01:~$ ps aux | grep fpm | grep master
root 5002 13.0 0.0 609140 10236 ? Ss 07:21 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
pmoust@webserver01:~$ sudo kill -USR2 5002
pmoust@webserver01:~$ ps aux | grep fpm | grep master
root 10771 10.6 0.0 609140 10228 ? Ss 07:26 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)

On 14.04 with ondrej's ppa
root@sandbox01:/home/pmoust# ps aux | grep fpm | grep master
root 21758 0.0 0.1 389924 23812 ? Ss 07:19 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
root@sandbox01:/home/pmoust# kill -1 21758
root@sandbox01:/home/pmoust# ps aux | grep fpm | grep master
root 21758 0.0 0.1 389924 23812 ? Ss 07:19 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
root@sandbox01:/home/pmoust# kill -USR2 21758
root@sandbox01:/home/pmoust# ps aux | grep fpm | grep master
root 21758 0.1 0.1 389924 23816 ? Ss 07:19 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)

[Ondřej Surý (ondrej)]

What does the log says?

# apt-cache policy php5-fpm
php5-fpm:
  Installed: 5.6.0~rc2+dfsg-1
  Candidate: 5.6.0~rc2+dfsg-1
  Version table:
 *** 5.6.0~rc2+dfsg-1 0
        700 http://ftp.cz.debian.org/debian/ sid/main amd64 Packages
        100 /var/lib/dpkg/status
     5.6.0~rc1+dfsg-1 0
        800 http://ftp.cz.debian.org/debian/ jessie/main amd64 Packages
     5.4.4-14+deb7u11 0
        900 http://security.debian.org/ wheezy/updates/main amd64 Packages
     5.4.4-14+deb7u9 0
        900 http://ftp.cz.debian.org/debian/ wheezy/main amd64 Packages

# tail -f /var/log/php5-fpm.log&

It works for all variants of reload command:

# systemctl reload php5-fpm
[07-Jul-2014 09:47:05] NOTICE: Reloading in progress ...
[07-Jul-2014 09:47:05] NOTICE: reloading: execvp("/usr/sbin/php5-fpm", {"/usr/sbin/php5-fpm", "--nodaemonize", "--fpm-config", "/etc/php5/fpm/php-fpm.conf"})
[07-Jul-2014 09:47:05] NOTICE: using inherited socket fd=7, "/var/run/php5-fpm.sock"
[07-Jul-2014 09:47:05] NOTICE: fpm is running, pid 14193
[07-Jul-2014 09:47:05] NOTICE: ready to handle connections
[07-Jul-2014 09:47:05] NOTICE: systemd monitor interval set to 10000ms

# kill -USR2 14193
# [07-Jul-2014 09:47:55] NOTICE: Reloading in progress ...
[07-Jul-2014 09:47:55] NOTICE: reloading: execvp("/usr/sbin/php5-fpm", {"/usr/sbin/php5-fpm", "--nodaemonize", "--fpm-config", "/etc/php5/fpm/php-fpm.conf"})
[07-Jul-2014 09:47:55] NOTICE: using inherited socket fd=7, "/var/run/php5-fpm.sock"
[07-Jul-2014 09:47:55] NOTICE: fpm is running, pid 14193
[07-Jul-2014 09:47:55] NOTICE: ready to handle connections
[07-Jul-2014 09:47:55] NOTICE: systemd monitor interval set to 10000ms

# kill -HUP 14193
# [07-Jul-2014 09:48:20] NOTICE: Reloading in progress ...
[07-Jul-2014 09:48:20] NOTICE: reloading: execvp("/usr/sbin/php5-fpm", {"/usr/sbin/php5-fpm", "--nodaemonize", "--fpm-config", "/etc/php5/fpm/php-fpm.conf"})
[07-Jul-2014 09:48:20] NOTICE: using inherited socket fd=7, "/var/run/php5-fpm.sock"
[07-Jul-2014 09:48:20] NOTICE: fpm is running, pid 14193
[07-Jul-2014 09:48:20] NOTICE: ready to handle connections
[07-Jul-2014 09:48:20] NOTICE: systemd monitor interval set to 10000ms

[Panagiotis Moustafellos (pmoust)]

Rather strange,

root@sandbox01:/home/pmoust# sudo service php5-fpm restart
php5-fpm stop/waiting
php5-fpm start/running, process 2193
root@sandbox01:/home/pmoust# tailf /var/log/php5-fpm.log &
[1] 2703
root@sandbox01:/home/pmoust# [07-Jul-2014 07:28:47] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful
root@sandbox01:/home/pmoust# apt-cache policy php5-fpm
php5-fpm:
  Installed: 5.5.14+dfsg-2+deb.sury.org~trusty+1
  Candidate: 5.5.14+dfsg-2+deb.sury.org~trusty+1
  Version table:
 *** 5.5.14+dfsg-2+deb.sury.org~trusty+1 0
        990 http://ppa.launchpad.net/ondrej/php5/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     5.5.9+dfsg-1ubuntu4.2 0
        990 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
        990 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages
     5.5.9+dfsg-1ubuntu4 0
        990 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
root@sandbox01:/home/pmoust# ps aux | grep fpm | grep master
root 2193 0.3 0.1 389924 23812 ? Ss 08:13 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)

[Panagiotis Moustafellos (pmoust)]

Did not paste the important part, here it goes

root@sandbox01:/home/pmoust# kill -HUP 2193; echo $?
0
root@sandbox01:/home/pmoust# kill -USR2 2193; echo $?
0
root@sandbox01:/home/pmoust# ps aux | grep fpm | grep master
root 2193 0.3 0.1 389924 23812 ? Ss 08:13 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)

[Panagiotis Moustafellos (pmoust)]

Shall I try fetching upstream and applying your patch or that's the one the ppa is built with?

[Ondřej Surý (ondrej)]

There's nothing wrong with master process NOT CHANGING the PID. I told you to look at the logs and not at the PID, so please do so before jumping to wrong conclusions, thanks.

[Panagiotis Moustafellos (pmoust)]

Hi Ondrej, sorry if I seemed pushy, not my intention.
The logs I pasted above refer to the exact test procedure you requested,
here are some fresh ones;

root@sandbox01:/home/pmoust# php -v
PHP 5.5.14-2+deb.sury.org~trusty+1 (cli) (built: Jul 2 2014 12:14:44)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
root@sandbox01:/home/pmoust# tailf /var/log/php5-fpm.log &
[2] 26256
root@sandbox01:/home/pmoust# [07-Jul-2014 09:49:25] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful

[07-Jul-2014 10:09:33] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful

[07-Jul-2014 10:37:59] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful

[07-Jul-2014 13:25:52] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful

[07-Jul-2014 15:40:58] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful

root@sandbox01:/home/pmoust# ps aux | grep fpm | grep master
root 8977 0.0 0.1 392060 23912 ? Ss Jul07 0:04 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
root@sandbox01:/home/pmoust# kill -HUP 8977
root@sandbox01:/home/pmoust# kill -USR2 8977
root@sandbox01:/home/pmoust#

Not fresh workers spawned.

[Ondřej Surý (ondrej)]

Did you modify the standard logging?

# grep -E 'log' /etc/php5/fpm/php-fpm.conf /etc/php5/fpm/pool.d/* | grep -v ":;"

?

[Panagiotis Moustafellos (pmoust)]

Not really,

pmoust@sandbox01:~⟫ grep -E 'log' /etc/php5/fpm/php-fpm.conf /etc/php5/fpm/pool.d/* | grep -v ":;"
/etc/php5/fpm/php-fpm.conf:error_log = /var/log/php5-fpm.log
/etc/php5/fpm/pool.d/main.conf:access.log = /var/log/fpm/main.access.log
/etc/php5/fpm/pool.d/main.conf:slowlog = /var/log/fpm/main.log.slow
/etc/php5/fpm/pool.d/main.conf:request_slowlog_timeout = 30s

[Ondřej Surý (ondrej)]

I have no idea why there's nothing in the logs, but I can say for sure that the PID of the master process *DOES NOT* change between reloads (the execve() call exchange the running code, but not the PID of the process).

The easy thing how to check if the pool.d/* processes do reload is to change some easily visible parameter like socket path and issue a reload and then check if the socket path has changed.

[Panagiotis Moustafellos (pmoust)]

I am aware what execve() call does.
So there must be an inconsistency between 5.3.x php fpm and the one under your ppa?
Centos 6.4;
[root@cygnus ~]# php -v
PHP 5.3.3 (cli) (built: Dec 11 2013 03:29:57)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
    with the ionCube PHP Loader v4.2.2, Copyright (c) 2002-2012, by ionCube Ltd., and
    with Xdebug v2.2.0rc1, Copyright (c) 2002-2012, by Derick Rethans
[root@cygnus ~]# service php-fpm status
php-fpm (pid 1334) is running...
[root@cygnus ~]# kill -USR2 1334
[root@cygnus ~]# service php-fpm status
php-fpm (pid 12361) is running...

Ubuntu 12.04
pmoust@webserver01:~$ ps aux | grep fpm | grep master
root 1202 13.0 0.0 609140 11236 ? Ss 09:19 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
pmoust@webserver01:~$ sudo kill -USR2 1202
pmoust@webserver01:~$ ps aux | grep fpm | grep master
root 5271 10.6 0.0 609140 11228 ? Ss 09:24 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)

But on php5-fpm of 14.04 under your ppa the master process of *fpm* does not change?

I'll check as soon as I am back at the office whether opcache is flushed/changed parameters are applied etc after issuing -1 or -USR2 using your PPA.

Thanks for your effort, my team and I have already voted in for your patch to be merged upstream.

[Ondřej Surý (ondrej)]

PHP 5.3 was first version to have php-fpm integrated, so I guess there might have been a room for improvement.

I haven't packaged FPM in Debian squeeze, so I cannot confirm the PHP 5.3 behaviour.

[Panagiotis Moustafellos (pmoust)]

@Ondrej, I can confirm that with your latest PPA the reload is fine. Thanks!

[Michael Shadle (mshadle)]

What I just noticed (on an Ubuntu 14.04.4 LTS, using ondrej's PPA) is that /etc/init.d/php5-fpm still exists.

When I issue "service php5-fpm reload" - my FPM engines disappear, even though I have /etc/init/php5-fpm.conf and /etc/init/php5-fpm.override set to enable the "reload" parameter.

Looks like when I run "service php5-fpm reload" it is trying to hit that (old, outdated) /etc/init.d/php5-fpm script, and wasn't hitting the upstart version.

Removed the old init file, and now "reload" works fine.

I am sure there are reasons for leaving it behind (such as if someone isn't using upstart?) but it causes a conflict if left in place.

/etc/init.d/php5-fpm might need some additional attention, to read from the same pid files, or whatever (?) if it doesn't get removed soon.

[Panagiotis Moustafellos (pmoust)]

It should be fixed upstream, simply adopting Ondrejs PPA *and* removing `/etc/init.d/php5-fpm` breaks BC and doesn't make much sense.
@deekshammhs10: Any ETA?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Reno Reckling (e-ubuntu-m)]

Any updates on this? This is causing us huge troubles because every configuration update via puppet triggers a service reload which kills the php5-fpm parent process and detaches it's clients, leading to stale code running in production after deployments.

[Robie Basak (racb)]

I have documented the workaround in the bug description. This bug presumably does not affect releases >= Vivid since it is related to upstart and Ubuntu runs systemd since Vivid. So marking Fix Released and creating a Trusty task in case someone can come up with a good fix for Trusty that can be SRU'd. Importance -> Medium as a workaround is available.

[Bryce Harrington (bryce)]

(Trusty is past end-of-standard-support.)