Please backport haproxy 1.5.14-1ubuntu0.15.10.1 from wily-updates

Bug #1494141 reported by Claudio Kuenzler on 2015-09-10
34
This bug affects 4 people
Affects Status Importance Assigned to Milestone
trusty-backports
Medium
Unassigned

Bug Description

On a new installation of Ubuntu 14.04.3 LTS I installed HAProxy 1.5 from trusty-backports (1.5.4-1ubuntu2.1~ubuntu14.04.1).

When I restarted HAProxy, I got random HTTP 503 although the backend servers were all working fine. By checking netstat, I saw that HAProxy was listening multiple times on the frontend ports.

It seems that the init script coming with the installation does not work correctly. The processes are not terminated correctly when using stop (or restart, in this matter, either).

Only with a kill I was able to correctly terminate the HAProxy processes.

The following output should show more clarity:

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4221/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# service haproxy stop
 * Stopping haproxy haproxy [ OK ]

root@mylinux:~# service haproxy status
haproxy not running.

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 4221/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4653/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 4221/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# killall haproxy

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# service haproxy start
 * Starting haproxy haproxy [ OK ]

root@mylinux:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 8205/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

root@mylinux:~# service haproxy restart
 * Restarting haproxy haproxy [ OK ]

root@nzzad-lb01-test:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8286/haproxy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 956/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 855/sshd
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 8286/haproxy
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 8205/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 8286/haproxy
tcp 0 0 0.0.0.0:8099 0.0.0.0:* LISTEN 8205/haproxy
tcp6 0 0 :::22 :::* LISTEN 855/sshd

description: updated
description: updated
Louis Bouchard (louis) on 2015-09-15
Changed in haproxy (Ubuntu):
assignee: nobody → Louis Bouchard (louis-bouchard)
Louis Bouchard (louis) on 2015-09-16
Changed in haproxy (Ubuntu):
status: New → In Progress
importance: Undecided → Medium
Louis Bouchard (louis) wrote :

Hi,

haproxy 1.5.4 introduces the following change :

        ret=0
- start-stop-daemon --quiet --oknodo --stop \
- --retry 5 --pidfile $PIDFILE --exec $HAPROXY || ret=$?
+ for pid in $(cat $PIDFILE); do
+ start-stop-daemon --quiet --oknodo --stop \
+ --retry 5 --pid $pid --exec $HAPROXY || ret=$?
+ done

start-stop-daemon --pidfile is replaced by --pid but this option is only available in dpkg 1.17.6 and onward :

$ man start-stop-daemon (on Wily) :
       [--pid] pid
              Check for a process with the specified pid (since version 1.17.6). The pid must be a number greater than 0.

This is not available on trusty, which explains why the process does not stop. I must admit that silently ignoring a false option is not an appropriate behavior though

The reason for that change is that, if nbproc > 1 in the haproxy config file, all the PIDs will be in one single /var/run/haproxy.pid file so in order to kill each process, we must loop through the pid list.

Louis Bouchard (louis) on 2015-09-17
affects: haproxy (Ubuntu) → trusty-backports
Louis Bouchard (louis) wrote :

Marking task Trusty as invalid, since haproxy 1.4 uses --pidfile and is unaffected.

Marking task Wily as fix released as the --pid option is available in Wily so haproxy 1.5 is unaffected

Changed in haproxy (Ubuntu):
status: New → Fix Released
Changed in haproxy (Ubuntu Trusty):
status: New → Invalid
Claudio Kuenzler (napsty) wrote :

Salut Louis,

Thanks for taking care of this bug. Have I understood correctly that the haproxy package in trusty-backports is already updated? Or do I have to wait until the package is released to all mirrors?

Louis Bouchard (louis) wrote :

Patch for the fix to the backport package

Louis Bouchard (louis) wrote :

Bonjour Claudio,

No, I have just submitted the fix for inclusion in the backported package. Once the backport team has taken care of it, it will be indicated as such in the bug.

Kind regards,

..Louis

Robie Basak (racb) wrote :

Louis,

I can't sponsor your debdiff into backports, but be careful of ordering issues in your patch. clean() should be defined before the trap is set, and tmp should be defined before any point that clean() could be called. In general you should quote "$tmp" as well in case it ends up with spaces (eg. if $TMPDIR has a space in it).

There's another haproxy backport request that should fix this (from utopic version). It was missing the testing of one of the reverse dependencies.

Micah, got the bug number ? Maybe we should make this one a duplicate then ?

..Louis

Micah Gersten (micahg) wrote :

Oh, I didn't realize that my previous backport upload was already accepted. This still needs a patch. I think we should just apply http://launchpadlibrarian.net/213125772/haproxy_1.4.24-2ubuntu0.1_1.4.24-2ubuntu0.2.diff.gz to backports, could you please test this? I've upload a test package to my PPA ppa:micahg/ppa

Louis Bouchard (louis) wrote :

Hello,

Unfortunately Jorge's patch is incomplete. The reason for the looping on pid is that, if nproc > 1, the pidfile will contain more than one PID and using --pidfile will only kill the first one.

The proposed patch here will fix both situation, i.e. the fact that --pid is silently ignored and that if nproc > 1, all processes will be killed.

I think we need to create new bug for Trusty as, if nproc > 1, only the first PID will be killed.

Claudio Kuenzler (napsty) wrote :

The "new bug" for trusty concerning if nproc > 1 is probably this one: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1481737 ?

Louis Bouchard (louis) wrote :

Good catch Claudio. The bug is present in Trusty and Trusty-backports.

Prior to LP: #1477198, the stop script did not stop at all because it relied on the --pid option that is introduced in dpkg version 1.17.6 as outlined in the man page of start-stop-daemon :

       [--pid] pid
              Check for a process with the specified pid (since version 1.17.6). The pid must be a number greater than 0.

Now with the fix introduced by LP: #1477198, it will work as expected unless nproc > 1. Otherwise, we will see the behavior described in LP: #1481737 whereas processes other than the first PID appearing in the pidfile will remain active.

The patch included here solves both situation as it loops through all the PIDs in the pidfile & terminates them using a temporary pidfile.

I will carry on this explanation in the other bug so the version in -updates is fixed as well.

This bug remains valid as we still see only the first PID being killed with the recent patch.

Louis Bouchard (louis) wrote :

FYI, I will rebase the debdiff on the new version currently in backport & repost.

Claudio Kuenzler (napsty) wrote :

By the way... this bug was also backported into the official trusty version of HAProxy (1.4).. Jeez!
In the meantime its fixed again.

haproxy (1.4.24-2ubuntu0.2) trusty; urgency=high

  * debian/haproxy.init:
    + Pass the pidfile to the --pidfile argument instead
      of the PID number. (LP: #1477198).

 -- Jorge Niedbalski <email address hidden> Wed, 22 Jul 2015 11:12:26 -0500

haproxy (1.4.24-2ubuntu0.1) trusty; urgency=medium

  * debian/haproxy.init:
     + Backport of utopic start/stop routines,
       uses start-stop-daemon to reliable terminate all
       haproxy processes and return the proper exit code.
       (LP: #1462495).
     + Backport of utopic check_haproxy_config routine,
       that test the configuration before start or reload.
       (LP: #1468879)

 -- Jorge Niedbalski <email address hidden> Thu, 25 Jun 2015 12:51:05 -0700

That's just as an additional note.

Louis Bouchard (louis) wrote :

Claudio, it is only partly fixed by Jorge's patch.

I have proposed the same fix than for this bug in LP: #1481737. Unfortunately, not being a core dev, it needs to be sponsored to be accepted. Then I can backport the fix here in 1.5

Sorry for the delay, I'll chase a sponsor.

Claudio Kuenzler (napsty) wrote :

Chase them with beer. Works faster ^^

Changed in haproxy (Ubuntu):
importance: Undecided → Medium
Changed in haproxy (Ubuntu Trusty):
importance: Undecided → Medium
Greg Gilbert (k-greg-n) wrote :

Just to throw in on this, I'm running 1.6.2 on 14.04.3 and it's not working here either. Specifically I see it running /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid, but /var/run/haproxy.pid doesn't exist.

Louis Bouchard (louis) wrote :

Greg, 1.6.2 is not packaged for Trusty. I don't know where it comes from.

Greg Gilbert (k-greg-n) wrote :

Ha, sorry about that - I crossed the wires in my head a bit. (ppa:vbernat/haproxy-1.6, for what it's worth.)

Anyway, should this work for 1.6?

Louis Bouchard (louis) wrote :

Vincent Bernat (vbernat) is the Debian maintainer for haproxy. His PPA assumes a version of dpkg that is newer than 1.17.7. I'll try to open a Debian bug on that.

The problem lies in the use of start-stop-daemon's new --pid option that is only available after wily.

If 1.6 is backported to any version earlier than wily, this bug will exist and the fix for it will need to be applied.

I just got the fix for 1.4 uploaded so I should be able to fix the 1.5 backport in the near future.

..Louis

no longer affects: haproxy (Ubuntu Trusty)
Louis Bouchard (louis) on 2015-11-27
Changed in trusty-backports:
assignee: Louis Bouchard (louis-bouchard) → nobody
tags: added: trusty
removed: haproxy
no longer affects: haproxy (Ubuntu)
Mark Casey (mark-casey) wrote :

I don't understand the current status of this bug.

I think what I'm reading is that this was fixed for trusty-backports inside of the work done to close #1481737, but I'm not certain. Is there anyone that could confirm that?

Thank you

Louis Bouchard (louis) wrote :

From what I can gather, the proper version that will fix this issue is the version which is in wily-updates :

http://archive.ubuntu.com/ubuntu/pool/main/h/haproxy/haproxy_1.5.14-1ubuntu0.15.10.1.dsc

Louis Bouchard (louis) on 2016-02-08
summary: - HAProxy 1.5 init script does not terminate processes
+ Please backport haproxy 1.5.14-1ubuntu0.15.10.1 from wily-updates
Louis Bouchard (louis) wrote :

Hello,

Here are the following verifications that were made on trusty :
 1) haproxy-1.5.14-1ubuntu0.15.10.1 builds correctly
 2) haproxy-1.5.14-1ubuntu0.15.10.1 installs correctly
 3) when nbproc > 1 is used, all haproxy processes are stopped correctly

I am now positive that 1.5.14 fixes the problem encountered with 1.5.4-1ubuntu2.1~ubuntu14.04.1 and that is also fixed in the latest 1.4 version available in -updates.

Iain Lane (laney) wrote :

backport accepted, enjoy

Changed in trusty-backports:
status: In Progress → Fix Released
Claudio Kuenzler (napsty) wrote :

Thanks for everyone involved to fix the bug. I can confirm stop/restart works as expected now with 1.5.14-1ubuntu0.15.10.1~ubuntu14.04.1 installed from backports on trusty.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers