HAProxy init script does not work correctly with nbproc configuration option

[Peter G. Marczis]

[SRU justification]
The sysVinit script does not completely stop processes when nbproc > 1

[Impact]
Stopping haproxy without this fix leaves all but one process running

[Fix]
Use temporary pidfile in a loop containing one single PID as the --pid option is not available for start-stop-daemon.

[Test Case]
Set nbproc > 4 in /etc/haproxy/haproxy.cfg
resart haproxy & verify that there is more than one haproxy process
Run :
$ ps aux | grep haproxy | grep -v grep | wc -l
4
$ service haproxy stop
$ ps aux | grep haproxy | grep -v grep | wc -l

Without the patch, the result of the last command will be 3. With the fix, it will be 0

[Regression]
The regression risk is in the use of a temporary file instead of the start-stop-daemon option.

It has been tested with TMPDIR="/tmp/My Dir" to check for spaces in directory names.

[Original description of the problem]
In case you have more than one process enabled in the haproxy config, the init script fails to stop all of those processes.
So to clarify you need to have this in your haproxy.cfg:

"""
...
global
        maxconn 32000
        ulimit-n 65536
        user haproxy
        group haproxy
        nbproc 2

...
"""

the problem is more visible if you set the nbproc to higher number.

service haproxy stop --> will only stop on of the haproxy processes.

The problem is so that start-stop-daemon can't handle pid files with multiple lines. Only stopping the first one. HAProxy does write all the pids started by itself into the pid file, so the problem is not in HAProxy, but in the start-stop-daemon or more likely in the init scrip of haproxy.
One solution or workaround is to remove pidfile option of start-stop-daemon in the init script, than it wil work as killall and will stop the haproxy processes properly.

To proof you can try this sequence:

service haproxy start #if its not running
ps ax | grep haproxy | grep -v grep | wc -l #this should report 2
service haproxy restart
ps ax | grep haproxy | grep -v grep | wc -l #this will report 3

The workaround as a diff:
root@ubi1:/opt# diff /etc/init.d/haproxy /etc/init.d/haproxy.orig
62c62
< --retry 5 --exec $HAPROXY || ret=$?
---
> --retry 5 --pidfile $PIDFILE --exec $HAPROXY || ret=$?

extra infos:
root@ubi1:/opt# lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04

root@ubi1:/opt# apt-cache policy haproxy
haproxy:
  Installed: 1.4.24-2ubuntu0.2
  Candidate: 1.4.24-2ubuntu0.2
  Version table:
     1.5.3-1~ubuntu14.04.1 0
        100 http://us.archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
 *** 1.4.24-2ubuntu0.2 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.24-2 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Peter G. Marczis (peter-marczis)]

Any additional info needed ?
I'm happy to help with anything !
Should be an upstart config for haproxy ? Or you plan to shift away from upstart ? (I remember to read that somewhere...)
Br,
 Peter.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

We're in sync with Debian on the haproxy package so this bug should be reported there. For Ubuntu, managing haproxy with systemd units would presumably be easiest; this could also be contributed to Debian.

Workaround: change /etc/init.d/haproxy as you described. Importance -> Medium since a workaround is available.

[Peter G. Marczis (peter-marczis)]

Is it really so that I should report the bug to debian via email ?
https://www.debian.org/Bugs/Reporting

Strange... please confirm.
Br,
 Peter.

[Robie Basak (racb)]

> Is it really so that I should report the bug to debian via email ?

Assuming the bug does reproduce on Debian and there isn't a bug filed there already, then yes - someone needs to in order to drive this bug towards a resolution. If you don't have time for this then that's fine - this bug can remain open until someone else gets to it.

[Peter G. Marczis (peter-marczis)]

No, no, I really want to get my hands dirty. Than first of all I will try to reproduce on Debian, and then check with them how it goes... In fact, should I use the newest mainstream version of Debian ? Or do you have any kind of mappings between Debian and Ubuntu releases ?

[Louis Bouchard (louis)]

Peter, you will not be able to reproduce on Debian unless you test versions prior to Stable.

The reason why the stop script does not stop at all processes is because it relied on the --pidfile option and this one only kills the first PID. When nproc > 1, the PIDFILE contains all PIDs. This is why debian introduced the use of --pid in a loop for all haproxy PIDs.

Now, the --pid option that is introduced in dpkg version 1.17.6 as outlined in the man page of start-stop-daemon :

       [--pid] pid
              Check for a process with the specified pid (since version 1.17.6). The pid must be a number greater than 0.

This version is available in all Debian releases since Stable but only available in Ubuntu Wily. A fix for this situation is available for haproxy 1.5 which is in trusty-backport (see LP: #1494141). I will backport that fix for the stable release and ask for an SRU on this one.

[Peter G. Marczis (peter-marczis)]

Hi Louis !
Yes, I have the same understanding of the problem ! So if the fix is available, and you do port it, I guess you can close the ticket :)
Thanks a lot for you work and effort !
Br,
 Peter.

[Mathieu Trudel-Lapierre (cyphermox)]

sponsored.

[James Page (james-page)]

Please can we ensure that this is actually fixed in the current development release as well; we're backporting haproxy to 14.04 as part of the Ubuntu Cloud Archive, so having backwards compat in the latest packages with 14.04 is important.

[James Page (james-page)]

Related bug 1477198

[James Page (james-page)]

There's a small niggle in the proposed change for trusty; the 'clean' exit handling will override the exit code from the init script, resulting in the return code being 0 all of the time; dropping the exit from within 'clean' will resolve this.

I've uploaded Louis' fix with this small improvement to xenial

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 1.6.2-2ubuntu3

---------------
haproxy (1.6.2-2ubuntu3) xenial; urgency=medium

  * d/haproxy.init: Ensure that cleanup does not override exit status
    of init script.

 -- James Page <email address hidden> Mon, 09 Nov 2015 14:35:15 +0000

[Chris J Arges (arges)]

Hello Peter, or anyone else affected,

Accepted haproxy into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.5.14-1ubuntu0.15.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 1.5.14-1ubuntu0.15.10.1

---------------
haproxy (1.5.14-1ubuntu0.15.10.1) wily; urgency=medium

  * Ensure that haproxy processes are terminated correctly when executing
    stop/restart operations, easing backports to pre-systemd versions of
    Ubuntu (LP: #1477198, #1481737).

 -- James Page <email address hidden> Mon, 09 Nov 2015 16:51:46 +0000

[Chris J Arges (arges)]

The verification of the Stable Release Update for haproxy has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Chris J Arges (arges)]

Hello Peter, or anyone else affected,

Accepted haproxy into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.4.24-2ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Can someone post an SRU for vivid? Thanks

[Louis Bouchard (louis)]

ok, I'm on the vivid SRU

[Louis Bouchard (louis)]

debdiff for vivid SRU

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 1.4.24-2ubuntu0.3

---------------
haproxy (1.4.24-2ubuntu0.3) trusty; urgency=medium

  * debian/haproxy.init:
    - Loops through all PIDs present in the PIDFILE when
      nproc > 1 (LP: #1481737)

 -- Louis Bouchard <email address hidden> Wed, 23 Sep 2015 17:45:41 +0200

[Chris J Arges (arges)]

Sponsored for vivid

[Brian Murray (brian-murray)]

Hello Peter, or anyone else affected,

Accepted haproxy into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.5.10-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ubuntu Foundations Team Bug Bot (crichton)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for vivid for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Martin Pitt (pitti)]

The version of haproxy in the proposed pocket of Vivid that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.