Stop doesn't work on 14.04 (start-stop-daemon --pid not supported)

Bug #1477198 reported by Jorge Niedbalski on 2015-07-22
56
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
High
Unassigned
Kilo
High
Unassigned
Liberty
High
James Page
trusty-backports
Undecided
Unassigned
haproxy (Debian)
Fix Released
Unknown
haproxy (Ubuntu)
High
James Page
Trusty
Critical
Unassigned
Wily
High
James Page
Xenial
High
James Page

Bug Description

[Description]

The stop method is not working properly. I removed the --oknodo && --quiet and is returning (No /usr/sbin/haproxy found running; none killed)

I think this is a regression caused by the incorporation of this lines on the stop method:

+ for pid in $(cat $PIDFILE); do
+ start-stop-daemon --quiet --oknodo --stop \
+ --retry 5 --pid $pid --exec $HAPROXY || ret=$?

root@juju-machine-1-lxc-0:~# service haproxy status
haproxy is running.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 1513 906 0 14:33 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2277 906 0 14:33 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2505 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2523 906 0 14:33 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy stop
 * Stopping haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2505 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2584 906 0 14:34 pts/6 00:00:00 grep --color=auto haproxy
root@juju-machine-1-lxc-0:~# service haproxy start
 * Starting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2505 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2591 1 0 14:34 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 2610 906 0 14:34 pts/6 00:00:00 grep --color=auto haproxy

[Impact]

- 'service stop/restart' doesn't works properly.

[Test Case]

- Install latest haproxy package.
- Run service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

- Run service haproxy stop

root@juju-machine-1-lxc-0:~# service haproxy stop
 * Stopping haproxy haproxy
   ...done.
root@juju-machine-1-lxc-0:~# ps -ef| grep haproxy
haproxy 1269 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy 2169 1 0 14:33 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

[Other info]

Related bugs:
 * bug 1481737: HAProxy init script does not work correctly with nbproc configuration option

description: updated
Jorge Niedbalski (niedbalski) wrote :
tags: added: sts
Chris J Arges (arges) on 2015-07-22
Changed in haproxy (Ubuntu):
status: New → Fix Released
Changed in haproxy (Ubuntu Trusty):
status: New → In Progress
assignee: nobody → Jorge Niedbalski (niedbalski)
importance: Undecided → Critical
Jorge Niedbalski (niedbalski) wrote :

The attached patch fixes the issue.

root@juju-testing-machine-15:/home/ubuntu# service haproxy start
 * Starting haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
haproxy 28338 1 0 15:29 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-15:/home/ubuntu# service haproxy stop
 * Stopping haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
root@juju-testing-machine-15:/home/ubuntu# service haproxy status
haproxy not running.
root@juju-testing-machine-15:/home/ubuntu# service haproxy start
 * Starting haproxy haproxy
                                                                                                                                                                  [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
haproxy 28398 1 0 15:29 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-15:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# ps -fea |grep haproxy
haproxy 28419 1 0 15:30 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-15:/home/ubuntu# service haproxy stop
 * Stopping haproxy haproxy [ OK ]
root@juju-testing-machine-15:/home/ubuntu# service haproxy status
haproxy not running.

description: updated

Hello Jorge, or anyone else affected,

Accepted haproxy into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.4.24-2ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed

I have verified that the -proposed package fixes the issue. Thanks.

root@juju-testing-machine-18:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
haproxy 8530 0.0 0.0 20300 636 ? Ss 19:47 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-18:/home/ubuntu# service haproxy stop
 * Stopping haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
root@juju-testing-machine-18:/home/ubuntu# service haproxy start
 * Starting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
haproxy 8567 0.0 0.0 20300 632 ? Ss 19:47 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root@juju-testing-machine-18:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# service haproxy restart
 * Restarting haproxy haproxy [ OK ]
root@juju-testing-machine-18:/home/ubuntu# ps aux|grep haproxy
haproxy 8607 0.0 0.0 20300 636 ? Ss 19:47 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
root 8611 0.0 0.0 10432 624 pts/0 S+ 19:47 0:00 grep --color=auto haproxy
root@juju-testing-machine-18:/home/ubuntu#

tags: added: verification-done
removed: verification-needed
David Ames (thedac) wrote :

Tested the -proposed package and it resolves an Openstack charms bug https://bugs.launchpad.net/charms/+source/nova-cloud-controller/+bug/1479416

Jorge Niedbalski (niedbalski) wrote :

Upgrades from trusty - > vivid, trusty -> utopic http://paste.ubuntu.com/11960633/ , all seems OK.

This bug was fixed in the package haproxy - 1.4.24-2ubuntu0.2

---------------
haproxy (1.4.24-2ubuntu0.2) trusty; urgency=high

  * debian/haproxy.init:
    + Pass the pidfile to the --pidfile argument instead
      of the PID number. (LP: #1477198).

 -- Jorge Niedbalski <email address hidden> Wed, 22 Jul 2015 11:12:26 -0500

Changed in haproxy (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for haproxy has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Scott Moser (smoser) on 2015-11-06
Changed in cloud-archive:
status: New → Confirmed
importance: Undecided → High
Changed in haproxy (Ubuntu):
importance: Undecided → Medium
Scott Moser (smoser) on 2015-11-06
description: updated
tags: added: kanban-cross-team
summary: - Stop doesn't works on Trusty
+ Stop doesn't work on Trusty

Kilo cloud archive does not contain haproxy backport - marking invalid.

James Page (james-page) wrote :

Liberty cloud archive includes a backport of haproxy from wily; I guess we need to pull in the stop patch as done for trusty.

Changed in haproxy (Ubuntu Trusty):
status: Fix Released → New
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in haproxy (Ubuntu Wily):
importance: Undecided → High
Changed in haproxy (Ubuntu Trusty):
importance: Critical → High
status: New → Fix Released
Changed in haproxy (Ubuntu Xenial):
status: Fix Released → New
Changed in haproxy (Ubuntu Trusty):
importance: High → Critical
Changed in haproxy (Ubuntu Xenial):
importance: Medium → High
James Page (james-page) wrote :

Marking for Xenial and Wily as well; although these release use systemd, they are the source for the cloud archive and I'd prefer not to carry a delta as it means we pickup any other fixes automatically including security updates.

James Page (james-page) wrote :

Uploads to xenial and wily with required patch made; subscribing SRU team for review.

James Page (james-page) wrote :

(patch also submitted back to Debian as well)

Changed in haproxy (Ubuntu Wily):
status: New → In Progress
Changed in haproxy (Ubuntu Xenial):
status: New → In Progress
Changed in haproxy (Ubuntu Wily):
assignee: nobody → James Page (james-page)
Changed in haproxy (Ubuntu Xenial):
assignee: nobody → James Page (james-page)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.6.2-2ubuntu1

---------------
haproxy (1.6.2-2ubuntu1) xenial; urgency=medium

  [ Jorge Niedbalski ]
  * debian/haproxy.init:
    + Pass the pidfile to the --pidfile argument instead of the PID
      number, easing backports to pre-systemd versions of Ubuntu
      and Debian (LP: #1477198).

 -- James Page <email address hidden> Mon, 09 Nov 2015 09:44:46 +0000

Changed in haproxy (Ubuntu Xenial):
status: In Progress → Fix Released
James Page (james-page) wrote :

Hmm - Vincent just pointed out that if nbproc > 1, then there are multiple pids to kill in the pidfile, which he thinks will not work with the patch applied in 14.04 and xenial, and proposed for SRU.

Investigating.

James Page (james-page) on 2015-11-09
summary: - Stop doesn't work on Trusty
+ Stop doesn't work on 14.04 (start-stop-daemon --pid not supported)
James Page (james-page) on 2015-11-09
Changed in haproxy (Ubuntu Xenial):
status: Fix Released → In Progress
Changed in haproxy (Debian):
status: Unknown → New
James Page (james-page) wrote :

Related bug 1481737

I think we'll try nail both of these in a single SRU for wily, and follow up on the original trusty sru for multi-proc configurations

James Page (james-page) on 2015-11-09
Changed in haproxy (Ubuntu Xenial):
status: In Progress → Fix Released

Hello Jorge, or anyone else affected,

Accepted haproxy into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.5.14-1ubuntu0.15.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Wily):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Andreas Hasenack (ahasenack) wrote :

This worked for me in trusty with the proposed cloud-archive repository.

Before, when I restarted haproxy, I got an extra process:
root@juju-machine-2-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.
root@juju-machine-2-lxc-0:~# ps fxaw|grep haproxy
  32215 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
  32547 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

Now it's just one as expected, replacing the one that was running before:

root@juju-machine-2-lxc-0:~# ps fxaw|grep haproxy
  33493 pts/3 S+ 0:00 \_ grep --color=auto haproxy
  33483 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 33115

root@juju-machine-2-lxc-0:~# service haproxy restart
 * Restarting haproxy haproxy
   ...done.

root@juju-machine-2-lxc-0:~# ps fxaw|grep haproxy
  33510 pts/3 S+ 0:00 \_ grep --color=auto haproxy
  33507 ? Ss 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

Using version 1.5.14-1ubuntu0.15.10.1~cloud0:
root@juju-machine-2-lxc-0:~# apt-cache policy haproxy
haproxy:
  Installed: 1.5.14-1ubuntu0.15.10.1~cloud0
  Candidate: 1.5.14-1ubuntu0.15.10.1~cloud0
  Version table:
 *** 1.5.14-1ubuntu0.15.10.1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-proposed/liberty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.5.14-1~cloud2 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/liberty/main amd64 Packages
     1.4.24-2ubuntu0.2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.4.24-2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

James Page (james-page) on 2015-11-10
tags: added: verification-done
removed: verification-needed
Changed in haproxy (Debian):
status: New → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.5.14-1ubuntu0.15.10.1

---------------
haproxy (1.5.14-1ubuntu0.15.10.1) wily; urgency=medium

  * Ensure that haproxy processes are terminated correctly when executing
    stop/restart operations, easing backports to pre-systemd versions of
    Ubuntu (LP: #1477198, #1481737).

 -- James Page <email address hidden> Mon, 09 Nov 2015 16:51:46 +0000

Changed in haproxy (Ubuntu Wily):
status: Fix Committed → Fix Released
Florian Haas (fghaas) wrote :

It appears that this issue is still present in the haproxy package currently available in trusty-backports (1.5.4-1ubuntu2.1~ubuntu14.04.1), so anyone using that package would still be affected by this issue.

Changed in haproxy (Debian):
status: Fix Committed → Fix Released
David Britton (davidpbritton) wrote :

Confirmed broken again on a default trusty install on maas:

haproxy:
  Installed: 1.5.4-1ubuntu2.1~ubuntu14.04.1
  Candidate: 1.5.4-1ubuntu2.1~ubuntu14.04.1
  Package pin: 1.5.4-1ubuntu2.1~ubuntu14.04.1
  Version table:
 *** 1.5.4-1ubuntu2.1~ubuntu14.04.1 500
        100 http://archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.24-2ubuntu0.4 500
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.4.24-2 500
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

James Page (james-page) wrote :

A fresh backport needs to be requested to resolve this in the backports package.

James Page (james-page) wrote :

David/Florian

Are you seeing the use of haproxy in backports in an OpenStack charm deployment? I can see one specific configuration option (prefer-ipv6) that would cause backports to be used, but in normal deployments the version in 14.04 should be used.

On Thu, Jan 28, 2016 at 07:39:59AM -0000, James Page wrote:
> David/Florian
>
> Are you seeing the use of haproxy in backports in an OpenStack charm
> deployment?

In the case of icehouse, I believe it would, yes. But I have not
tested.

> I can see one specific configuration option (prefer-ipv6)
> that would cause backports to be used, but in normal deployments the
> version in 14.04 should be used.

I hit this by simply deploying the haproxy charm on trusty. Openstack
was not really involved in what I was testing. I then recognized the
pattern of the bug.

--
David Britton <email address hidden>

James Page (james-page) wrote :

Icehouse deployment - confirmed not using backports

$ apt-cache policy haproxy
haproxy:
  Installed: 1.4.24-2ubuntu0.4
  Candidate: 1.4.24-2ubuntu0.4
  Version table:
     1.5.4-1ubuntu2.1~ubuntu14.04.1 0
        100 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
 *** 1.4.24-2ubuntu0.4 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.24-2 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Changed in haproxy (Juju Charms Collection):
status: New → Invalid
no longer affects: haproxy (Juju Charms Collection)
Jay R. Wren (evarlast) wrote :

using 1.5.10-1~0.IS.14.04.0 from `deb http://archive.admin.canonical.com/ubuntu trusty-cat-haproxy main` and haproxy juju charm configured with source to that apt list line.

Observing this behavior.

Part of the problem seems to be `service haproxy reload` uses haproxy -sf, but the pids listed as -sf arguents are not finishing and exiting. They continue to service requests with the old haproxy config.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.