Please backport haproxy 1.5.14-1ubuntu0.15.10.1 from wily-updates

[Claudio Kuenzler]

On a new installation of Ubuntu 14.04.3 LTS I installed HAProxy 1.5 from trusty-backports (1.5.4-1ubuntu2.1~ubuntu14.04.1).

When I restarted HAProxy, I got random HTTP 503 although the backend servers were all working fine. By checking netstat, I saw that HAProxy was listening multiple times on the frontend ports.

It seems that the init script coming with the installation does not work correctly. The processes are not terminated correctly when using stop (or restart, in this matter, either).

Only with a kill I was able to correctly terminate the HAProxy processes.

The following output should show more clarity:

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4221/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# service haproxy stop
 * Stopping haproxy haproxy [ OK ]

root@mylinux:~# service haproxy status
haproxy not running.

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4221/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# killall haproxy

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# service haproxy start
 * Starting haproxy haproxy [ OK ]

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 8205/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# service haproxy restart
 * Restarting haproxy haproxy [ OK ]

root@nzzad-lb01-test:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8286/haproxy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 8286/haproxy
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 8286/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 8205/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

[Louis Bouchard (louis)]

Hi,

haproxy 1.5.4 introduces the following change :

        ret=0
- start-stop-daemon --quiet --oknodo --stop \
- --retry 5 --pidfile $PIDFILE --exec $HAPROXY || ret=$?
+ for pid in $(cat $PIDFILE); do
+ start-stop-daemon --quiet --oknodo --stop \
+ --retry 5 --pid $pid --exec $HAPROXY || ret=$?
+ done

start-stop-daemon --pidfile is replaced by --pid but this option is only available in dpkg 1.17.6 and onward :

$ man start-stop-daemon (on Wily) :
       [--pid] pid
              Check for a process with the specified pid (since version 1.17.6). The pid must be a number greater than 0.

This is not available on trusty, which explains why the process does not stop. I must admit that silently ignoring a false option is not an appropriate behavior though

The reason for that change is that, if nbproc > 1 in the haproxy config file, all the PIDs will be in one single /var/run/haproxy.pid file so in order to kill each process, we must loop through the pid list.

[Louis Bouchard (louis)]

Marking task Trusty as invalid, since haproxy 1.4 uses --pidfile and is unaffected.

Marking task Wily as fix released as the --pid option is available in Wily so haproxy 1.5 is unaffected

[Claudio Kuenzler (napsty)]

Salut Louis,

Thanks for taking care of this bug. Have I understood correctly that the haproxy package in trusty-backports is already updated? Or do I have to wait until the package is released to all mirrors?

[Louis Bouchard (louis)]

Patch for the fix to the backport package

[Louis Bouchard (louis)]

Bonjour Claudio,

No, I have just submitted the fix for inclusion in the backported package. Once the backport team has taken care of it, it will be indicated as such in the bug.

Kind regards,

..Louis

[Robie Basak (racb)]

Louis,

I can't sponsor your debdiff into backports, but be careful of ordering issues in your patch. clean() should be defined before the trap is set, and tmp should be defined before any point that clean() could be called. In general you should quote "$tmp" as well in case it ends up with spaces (eg. if $TMPDIR has a space in it).

[Micah Gersten (micahg)]

There's another haproxy backport request that should fix this (from utopic version). It was missing the testing of one of the reverse dependencies.

[Louis Bouchard (louis)]

Micah, got the bug number ? Maybe we should make this one a duplicate then ?

..Louis

[Micah Gersten (micahg)]

Oh, I didn't realize that my previous backport upload was already accepted. This still needs a patch. I think we should just apply http://launchpadlibrarian.net/213125772/haproxy_1.4.24-2ubuntu0.1_1.4.24-2ubuntu0.2.diff.gz to backports, could you please test this? I've upload a test package to my PPA ppa:micahg/ppa

[Louis Bouchard (louis)]

Hello,

Unfortunately Jorge's patch is incomplete. The reason for the looping on pid is that, if nproc > 1, the pidfile will contain more than one PID and using --pidfile will only kill the first one.

The proposed patch here will fix both situation, i.e. the fact that --pid is silently ignored and that if nproc > 1, all processes will be killed.

I think we need to create new bug for Trusty as, if nproc > 1, only the first PID will be killed.

[Claudio Kuenzler (napsty)]

The "new bug" for trusty concerning if nproc > 1 is probably this one: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1481737 ?

[Louis Bouchard (louis)]

Good catch Claudio. The bug is present in Trusty and Trusty-backports.

Prior to LP: #1477198, the stop script did not stop at all because it relied on the --pid option that is introduced in dpkg version 1.17.6 as outlined in the man page of start-stop-daemon :

       [--pid] pid
              Check for a process with the specified pid (since version 1.17.6). The pid must be a number greater than 0.

Now with the fix introduced by LP: #1477198, it will work as expected unless nproc > 1. Otherwise, we will see the behavior described in LP: #1481737 whereas processes other than the first PID appearing in the pidfile will remain active.

The patch included here solves both situation as it loops through all the PIDs in the pidfile & terminates them using a temporary pidfile.

I will carry on this explanation in the other bug so the version in -updates is fixed as well.

This bug remains valid as we still see only the first PID being killed with the recent patch.

[Louis Bouchard (louis)]

FYI, I will rebase the debdiff on the new version currently in backport & repost.

[Claudio Kuenzler (napsty)]

By the way... this bug was also backported into the official trusty version of HAProxy (1.4).. Jeez!
In the meantime its fixed again.

haproxy (1.4.24-2ubuntu0.2) trusty; urgency=high

  * debian/haproxy.init:
    + Pass the pidfile to the --pidfile argument instead
      of the PID number. (LP: #1477198).

 -- Jorge Niedbalski <email address hidden> Wed, 22 Jul 2015 11:12:26 -0500

haproxy (1.4.24-2ubuntu0.1) trusty; urgency=medium

  * debian/haproxy.init:
     + Backport of utopic start/stop routines,
       uses start-stop-daemon to reliable terminate all
       haproxy processes and return the proper exit code.
       (LP: #1462495).
     + Backport of utopic check_haproxy_config routine,
       that test the configuration before start or reload.
       (LP: #1468879)

 -- Jorge Niedbalski <email address hidden> Thu, 25 Jun 2015 12:51:05 -0700

That's just as an additional note.

[Louis Bouchard (louis)]

Claudio, it is only partly fixed by Jorge's patch.

I have proposed the same fix than for this bug in LP: #1481737. Unfortunately, not being a core dev, it needs to be sponsored to be accepted. Then I can backport the fix here in 1.5

Sorry for the delay, I'll chase a sponsor.

[Claudio Kuenzler (napsty)]

Chase them with beer. Works faster ^^

[Greg Gilbert (k-greg-n)]

Just to throw in on this, I'm running 1.6.2 on 14.04.3 and it's not working here either. Specifically I see it running /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid, but /var/run/haproxy.pid doesn't exist.

[Louis Bouchard (louis)]

Greg, 1.6.2 is not packaged for Trusty. I don't know where it comes from.

[Greg Gilbert (k-greg-n)]

Ha, sorry about that - I crossed the wires in my head a bit. (ppa:vbernat/haproxy-1.6, for what it's worth.)

Anyway, should this work for 1.6?

[Louis Bouchard (louis)]

Vincent Bernat (vbernat) is the Debian maintainer for haproxy. His PPA assumes a version of dpkg that is newer than 1.17.7. I'll try to open a Debian bug on that.

The problem lies in the use of start-stop-daemon's new --pid option that is only available after wily.

If 1.6 is backported to any version earlier than wily, this bug will exist and the fix for it will need to be applied.

I just got the fix for 1.4 uploaded so I should be able to fix the 1.5 backport in the near future.

..Louis

[Mark Casey (mark-casey)]

I don't understand the current status of this bug.

I think what I'm reading is that this was fixed for trusty-backports inside of the work done to close #1481737, but I'm not certain. Is there anyone that could confirm that?

Thank you

[Louis Bouchard (louis)]

From what I can gather, the proper version that will fix this issue is the version which is in wily-updates :

http://archive.ubuntu.com/ubuntu/pool/main/h/haproxy/haproxy_1.5.14-1ubuntu0.15.10.1.dsc

[Louis Bouchard (louis)]

Hello,

Here are the following verifications that were made on trusty :
 1) haproxy-1.5.14-1ubuntu0.15.10.1 builds correctly
 2) haproxy-1.5.14-1ubuntu0.15.10.1 installs correctly
 3) when nbproc > 1 is used, all haproxy processes are stopped correctly

I am now positive that 1.5.14 fixes the problem encountered with 1.5.4-1ubuntu2.1~ubuntu14.04.1 and that is also fixed in the latest 1.4 version available in -updates.

[Iain Lane (laney)]

backport accepted, enjoy

[Claudio Kuenzler (napsty)]

Thanks for everyone involved to fix the bug. I can confirm stop/restart works as expected now with 1.5.14-1ubuntu0.15.10.1~ubuntu14.04.1 installed from backports on trusty.