Stop doesn't work on 14.04 (start-stop-daemon --pid not supported)

[Jorge Niedbalski]

[Description]

The stop method is not working properly. I removed the --oknodo && --quiet and is returning (No /usr/sbin/haproxy found running; none killed)

I think this is a regression caused by the incorporation of this lines on the stop method:

+ for pid in $(cat $PIDFILE); do
+ start-stop-daemon --quiet --oknodo --stop \
+ --retry 5 --pid $pid --exec $HAPROXY || ret=$?

root@juju-machine-1-lxc-0:~# service haproxy status
haproxy is running.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 1513 906 0 14:33 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2277 906 0 14:33 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2505 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2523 906 0 14:33 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy stop
 * Stopping haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2505 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2584 906 0 14:34 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy start
 * Starting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2505 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2591 1 0 14:34 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2610 906 0 14:34 pts/6 00:00:00 grep --color=auto haproxy

[Impact]

- 'service stop/restart' doesn't works properly.

[Test Case]

- Install latest haproxy package.
- Run service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

- Run service haproxy stop

root@juju-machine-1-lxc-0:~# service haproxy stop
 * Stopping haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

[Other info]

Related bugs:
 * bug 1481737: HAProxy init script does not work correctly with nbproc configuration option

[Jorge Niedbalski (niedbalski)]

The attached patch fixes the issue.

root@juju-testing-machine-15:/home/ubuntu# service haproxy start
 * Starting haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
haproxy 28338 1 0 15:29 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-15:/home/ubuntu# service haproxy stop
 * Stopping haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
root@juju-testing-machine-15:/home/ubuntu# service haproxy status
haproxy not running.
root@juju-testing-machine-15:/home/ubuntu# service haproxy start
 * Starting haproxy haproxy
                                                                                                                                                                  [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
haproxy 28398 1 0 15:29 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-15:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
haproxy 28419 1 0 15:30 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-15:/home/ubuntu# service haproxy stop
 * Stopping haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# service haproxy status
haproxy not running.

[Brian Murray (brian-murray)]

Hello Jorge, or anyone else affected,

Accepted haproxy into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.4.24-2ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Jorge Niedbalski (niedbalski)]

I have verified that the -proposed package fixes the issue. Thanks.

root@juju-testing-machine-18:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
haproxy 8530 0.0 0.0 20300 636 ? Ss 19:47 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-18:/home/ubuntu# service haproxy stop
 * Stopping haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
root@juju-testing-machine-18:/home/ubuntu# service haproxy start
 * Starting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
haproxy 8567 0.0 0.0 20300 632 ? Ss 19:47 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-18:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
haproxy 8607 0.0 0.0 20300 636 ? Ss 19:47 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 8611 0.0 0.0 10432 624 pts/0 S+ 19:47 0:00 grep --color=auto haproxy
root@juju-testing-machine-18:/home/ubuntu#

[David Ames (thedac)]

Tested the -proposed package and it resolves an Openstack charms bug https://bugs.launchpad.net/charms/+source/nova-cloud-controller/+bug/1479416

[Jorge Niedbalski (niedbalski)]

Upgrades from trusty - > vivid, trusty -> utopic http://paste.ubuntu.com/11960633/ , all seems OK.

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 1.4.24-2ubuntu0.2

---------------
haproxy (1.4.24-2ubuntu0.2) trusty; urgency=high

  * debian/haproxy.init:
    + Pass the pidfile to the --pidfile argument instead
      of the PID number. (LP: #1477198).

 -- Jorge Niedbalski <email address hidden> Wed, 22 Jul 2015 11:12:26 -0500

[Chris J Arges (arges)]

The verification of the Stable Release Update for haproxy has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

Kilo cloud archive does not contain haproxy backport - marking invalid.

[James Page (james-page)]

Liberty cloud archive includes a backport of haproxy from wily; I guess we need to pull in the stop patch as done for trusty.

[James Page (james-page)]

Marking for Xenial and Wily as well; although these release use systemd, they are the source for the cloud archive and I'd prefer not to carry a delta as it means we pickup any other fixes automatically including security updates.

[James Page (james-page)]

Uploads to xenial and wily with required patch made; subscribing SRU team for review.

[James Page (james-page)]

(patch also submitted back to Debian as well)

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 1.6.2-2ubuntu1

---------------
haproxy (1.6.2-2ubuntu1) xenial; urgency=medium

  [ Jorge Niedbalski ]
  * debian/haproxy.init:
    + Pass the pidfile to the --pidfile argument instead of the PID
      number, easing backports to pre-systemd versions of Ubuntu
      and Debian (LP: #1477198).

 -- James Page <email address hidden> Mon, 09 Nov 2015 09:44:46 +0000

[James Page (james-page)]

Hmm - Vincent just pointed out that if nbproc > 1, then there are multiple pids to kill in the pidfile, which he thinks will not work with the patch applied in 14.04 and xenial, and proposed for SRU.

Investigating.

[James Page (james-page)]

Related bug 1481737

I think we'll try nail both of these in a single SRU for wily, and follow up on the original trusty sru for multi-proc configurations

[Chris J Arges (arges)]

Hello Jorge, or anyone else affected,

Accepted haproxy into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.5.14-1ubuntu0.15.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andreas Hasenack (ahasenack)]

This worked for me in trusty with the proposed cloud-archive repository.

Before, when I restarted haproxy, I got an extra process:
root@juju-machine-2-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-2-lxc-0:~# ps fxaw|grep haproxy
  32215 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
  32547 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

Now it's just one as expected, replacing the one that was running before:

root@juju-machine-2-lxc-0:~# ps fxaw|grep haproxy
  33493 pts/3 S+ 0:00 \_ grep --color=auto haproxy
  33483 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 33115

root@juju-machine-2-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.

root@juju-machine-2-lxc-0:~# ps fxaw|grep haproxy
  33510 pts/3 S+ 0:00 \_ grep --color=auto haproxy
  33507 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

Using version 1.5.14-1ubuntu0.15.10.1~cloud0:
root@juju-machine-2-lxc-0:~# apt-cache policy haproxy
haproxy:
  Installed: 1.5.14-1ubuntu0.15.10.1~cloud0
  Candidate: 1.5.14-1ubuntu0.15.10.1~cloud0
  Version table:
 *** 1.5.14-1ubuntu0.15.10.1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-proposed/liberty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.5.14-1~cloud2 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/liberty/main amd64 Packages
     1.4.24-2ubuntu0.2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.4.24-2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 1.5.14-1ubuntu0.15.10.1

---------------
haproxy (1.5.14-1ubuntu0.15.10.1) wily; urgency=medium

  * Ensure that haproxy processes are terminated correctly when executing
    stop/restart operations, easing backports to pre-systemd versions of
    Ubuntu (LP: #1477198, #1481737).

 -- James Page <email address hidden> Mon, 09 Nov 2015 16:51:46 +0000

[Florian Haas (fghaas)]

It appears that this issue is still present in the haproxy package currently available in trusty-backports (1.5.4-1ubuntu2.1~ubuntu14.04.1), so anyone using that package would still be affected by this issue.

[David Britton (dpb)]

Confirmed broken again on a default trusty install on maas:

haproxy:
  Installed: 1.5.4-1ubuntu2.1~ubuntu14.04.1
  Candidate: 1.5.4-1ubuntu2.1~ubuntu14.04.1
  Package pin: 1.5.4-1ubuntu2.1~ubuntu14.04.1
  Version table:
 *** 1.5.4-1ubuntu2.1~ubuntu14.04.1 500
        100 http://archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.24-2ubuntu0.4 500
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.4.24-2 500
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[James Page (james-page)]

A fresh backport needs to be requested to resolve this in the backports package.

[James Page (james-page)]

David/Florian

Are you seeing the use of haproxy in backports in an OpenStack charm deployment? I can see one specific configuration option (prefer-ipv6) that would cause backports to be used, but in normal deployments the version in 14.04 should be used.

[David Britton (dpb)]

On Thu, Jan 28, 2016 at 07:39:59AM -0000, James Page wrote:
> David/Florian
>
> Are you seeing the use of haproxy in backports in an OpenStack charm
> deployment?

In the case of icehouse, I believe it would, yes. But I have not
tested.

> I can see one specific configuration option (prefer-ipv6)
> that would cause backports to be used, but in normal deployments the
> version in 14.04 should be used.

I hit this by simply deploying the haproxy charm on trusty. Openstack
was not really involved in what I was testing. I then recognized the
pattern of the bug.

--
David Britton <email address hidden>

[James Page (james-page)]

Icehouse deployment - confirmed not using backports

$ apt-cache policy haproxy
haproxy:
  Installed: 1.4.24-2ubuntu0.4
  Candidate: 1.4.24-2ubuntu0.4
  Version table:
     1.5.4-1ubuntu2.1~ubuntu14.04.1 0
        100 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
 *** 1.4.24-2ubuntu0.4 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.24-2 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[Jay R. Wren (evarlast)]

using 1.5.10-1~0.IS.14.04.0 from `deb http://archive.admin.canonical.com/ubuntu trusty-cat-haproxy main` and haproxy juju charm configured with source to that apt list line.

Observing this behavior.

Part of the problem seems to be `service haproxy reload` uses haproxy -sf, but the pids listed as -sf arguents are not finishing and exiting. They continue to service requests with the old haproxy config.