| 2019-10-20 02:01:29 |
James Troup |
bug |
|
|
added bug |
| 2019-10-20 02:01:40 |
James Troup |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2019-10-20 07:03:35 |
Launchpad Janitor |
haproxy (Ubuntu): status |
New |
Confirmed |
|
| 2019-10-20 19:51:43 |
Paul Collins |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2019-10-31 05:26:18 |
Haw Loeung |
bug |
|
|
added subscriber Haw Loeung |
| 2019-11-01 08:00:04 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Server |
| 2019-11-01 08:00:34 |
Christian Ehrhardt |
tags |
|
server-next |
|
| 2019-11-01 12:45:46 |
Simon Déziel |
bug |
|
|
added subscriber Simon Déziel |
| 2019-11-06 17:56:47 |
Andreas Hasenack |
haproxy (Ubuntu): importance |
Undecided |
High |
|
| 2019-11-06 17:56:49 |
Andreas Hasenack |
haproxy (Ubuntu): status |
Confirmed |
Triaged |
|
| 2019-11-12 12:09:51 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-11-12 12:09:51 |
Christian Ehrhardt |
bug task added |
|
haproxy (Ubuntu Bionic) |
|
| 2019-11-12 12:09:57 |
Christian Ehrhardt |
haproxy (Ubuntu Bionic): status |
New |
Triaged |
|
| 2019-11-12 12:09:59 |
Christian Ehrhardt |
haproxy (Ubuntu Bionic): importance |
Undecided |
High |
|
| 2019-11-12 12:10:02 |
Christian Ehrhardt |
haproxy (Ubuntu): status |
Triaged |
Fix Released |
|
| 2019-11-12 12:24:48 |
Christian Ehrhardt |
description |
On a Bionic/Stein cloud, after a network partition, we saw several units (glance, swift-proxy and cinder) fail to start haproxy, like so:
root@juju-df624b-6-lxd-4:~# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-10-20 00:23:18 UTC; 1h 35min ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 2002655 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143)
Process: 2002649 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 2002655 (code=exited, status=143)
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Starting HAProxy Load Balancer...
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Started HAProxy Load Balancer.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopping HAProxy Load Balancer...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : Exiting Master process...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [ALERT] 292/001652 (2002655) : Current worker 2002661 exited with code 143
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : All workers exited. Exiting... (143)
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Main process exited, code=exited, status=143/n/a
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopped HAProxy Load Balancer.
root@juju-df624b-6-lxd-4:~#
The Debian maintainer came up with the following patch for this:
https://www.mail-archive.com/haproxy@formilux.org/msg30477.html
Which was added to the 1.8.10-1 Debian upload and merged into upstream 1.8.13.
Unfortunately Bionic is on 1.8.8-1ubuntu0.4 and doesn't have this patch.
Please consider pulling this patch into an SRU for Bionic. |
[Impact]
* The master process will exit with the status of the last worker.
When the worker is killed with SIGTERM, it is expected to get 143 as an
exit status. Therefore, we consider this exit status as normal from a
systemd point of view. If it happens when not stopping, the systemd
unit is configured to always restart, so it has no adverse effect.
* Backport upstream fix - adding another accepted RC to the systemd
service
[Test Case]
* You want to install haproxy and have it running. Then sigterm it a lot.
With the fix it would restart the service all the time, well except
restart limit. But in the bad case it will just stay down and didn't
even try to restart it.
$ apt install haproxy
$ for x in {1..100}; do pkill -TERM -x haproxy ; sleep 0.1 ; done
$ systemctl status haproxy
[Regression Potential]
* This eventually is a conffile modification, so if there are other
modifications done by the user they will get a prompt. But that isn't a
regression. I checked the code and I can't think of another RC=143 that
would due to that "no more" detected as error. I really think other
than the update itself triggering a restart (as usual for services)
there is no further regression potential to this.
[Other Info]
* Fix already active in IS hosted cloud without issues since a while
* Also reports (comment #5) show that others use this in production as
well
---
On a Bionic/Stein cloud, after a network partition, we saw several units (glance, swift-proxy and cinder) fail to start haproxy, like so:
root@juju-df624b-6-lxd-4:~# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-10-20 00:23:18 UTC; 1h 35min ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 2002655 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143)
Process: 2002649 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 2002655 (code=exited, status=143)
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Starting HAProxy Load Balancer...
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Started HAProxy Load Balancer.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopping HAProxy Load Balancer...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : Exiting Master process...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [ALERT] 292/001652 (2002655) : Current worker 2002661 exited with code 143
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : All workers exited. Exiting... (143)
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Main process exited, code=exited, status=143/n/a
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopped HAProxy Load Balancer.
root@juju-df624b-6-lxd-4:~#
The Debian maintainer came up with the following patch for this:
https://www.mail-archive.com/haproxy@formilux.org/msg30477.html
Which was added to the 1.8.10-1 Debian upload and merged into upstream 1.8.13.
Unfortunately Bionic is on 1.8.8-1ubuntu0.4 and doesn't have this patch.
Please consider pulling this patch into an SRU for Bionic. |
|
| 2019-11-12 12:25:53 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/haproxy/+git/haproxy/+merge/375433 |
|
| 2019-11-18 07:57:14 |
Christian Ehrhardt |
description |
[Impact]
* The master process will exit with the status of the last worker.
When the worker is killed with SIGTERM, it is expected to get 143 as an
exit status. Therefore, we consider this exit status as normal from a
systemd point of view. If it happens when not stopping, the systemd
unit is configured to always restart, so it has no adverse effect.
* Backport upstream fix - adding another accepted RC to the systemd
service
[Test Case]
* You want to install haproxy and have it running. Then sigterm it a lot.
With the fix it would restart the service all the time, well except
restart limit. But in the bad case it will just stay down and didn't
even try to restart it.
$ apt install haproxy
$ for x in {1..100}; do pkill -TERM -x haproxy ; sleep 0.1 ; done
$ systemctl status haproxy
[Regression Potential]
* This eventually is a conffile modification, so if there are other
modifications done by the user they will get a prompt. But that isn't a
regression. I checked the code and I can't think of another RC=143 that
would due to that "no more" detected as error. I really think other
than the update itself triggering a restart (as usual for services)
there is no further regression potential to this.
[Other Info]
* Fix already active in IS hosted cloud without issues since a while
* Also reports (comment #5) show that others use this in production as
well
---
On a Bionic/Stein cloud, after a network partition, we saw several units (glance, swift-proxy and cinder) fail to start haproxy, like so:
root@juju-df624b-6-lxd-4:~# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-10-20 00:23:18 UTC; 1h 35min ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 2002655 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143)
Process: 2002649 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 2002655 (code=exited, status=143)
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Starting HAProxy Load Balancer...
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Started HAProxy Load Balancer.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopping HAProxy Load Balancer...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : Exiting Master process...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [ALERT] 292/001652 (2002655) : Current worker 2002661 exited with code 143
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : All workers exited. Exiting... (143)
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Main process exited, code=exited, status=143/n/a
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopped HAProxy Load Balancer.
root@juju-df624b-6-lxd-4:~#
The Debian maintainer came up with the following patch for this:
https://www.mail-archive.com/haproxy@formilux.org/msg30477.html
Which was added to the 1.8.10-1 Debian upload and merged into upstream 1.8.13.
Unfortunately Bionic is on 1.8.8-1ubuntu0.4 and doesn't have this patch.
Please consider pulling this patch into an SRU for Bionic. |
[Impact]
* The master process will exit with the status of the last worker.
When the worker is killed with SIGTERM, it is expected to get 143 as an
exit status. Therefore, we consider this exit status as normal from a
systemd point of view. If it happens when not stopping, the systemd
unit is configured to always restart, so it has no adverse effect.
* Backport upstream fix - adding another accepted RC to the systemd
service
[Test Case]
* You want to install haproxy and have it running. Then sigterm it a lot.
With the fix it would restart the service all the time, well except
restart limit. But in the bad case it will just stay down and didn't
even try to restart it.
$ apt install haproxy
$ for x in {1..100}; do pkill -TERM -x haproxy ; sleep 0.1 ; done
$ systemctl status haproxy
The above is a hacky way to trigger some A/B behavior on the fix.
It isn't perfect as systemd restart counters will kick in and you
essentially check a secondary symptom.
I'd recommend to in addition run the following:
$ apt install haproxy
$ for x in {1..1000}; do pkill -TERM -x haproxy ; sleep 0.001 systemctl
reset-failed haproxy.service; done
$ systemctl status haproxy
You can do so with even smaller sleeps, that should keep the service up
and running (this isn't changing with the fix, but should work with the new code).
[Regression Potential]
* This eventually is a conffile modification, so if there are other
modifications done by the user they will get a prompt. But that isn't a
regression. I checked the code and I can't think of another RC=143 that
would due to that "no more" detected as error. I really think other
than the update itself triggering a restart (as usual for services)
there is no further regression potential to this.
[Other Info]
* Fix already active in IS hosted cloud without issues since a while
* Also reports (comment #5) show that others use this in production as
well
---
On a Bionic/Stein cloud, after a network partition, we saw several units (glance, swift-proxy and cinder) fail to start haproxy, like so:
root@juju-df624b-6-lxd-4:~# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-10-20 00:23:18 UTC; 1h 35min ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 2002655 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143)
Process: 2002649 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 2002655 (code=exited, status=143)
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Starting HAProxy Load Balancer...
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Started HAProxy Load Balancer.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopping HAProxy Load Balancer...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : Exiting Master process...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [ALERT] 292/001652 (2002655) : Current worker 2002661 exited with code 143
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652 (2002655) : All workers exited. Exiting... (143)
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Main process exited, code=exited, status=143/n/a
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopped HAProxy Load Balancer.
root@juju-df624b-6-lxd-4:~#
The Debian maintainer came up with the following patch for this:
https://www.mail-archive.com/haproxy@formilux.org/msg30477.html
Which was added to the 1.8.10-1 Debian upload and merged into upstream 1.8.13.
Unfortunately Bionic is on 1.8.8-1ubuntu0.4 and doesn't have this patch.
Please consider pulling this patch into an SRU for Bionic. |
|
| 2019-11-22 11:52:58 |
Timo Aaltonen |
haproxy (Ubuntu Bionic): status |
Triaged |
Fix Committed |
|
| 2019-11-22 11:53:00 |
Timo Aaltonen |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-11-22 11:53:03 |
Timo Aaltonen |
bug |
|
|
added subscriber SRU Verification |
| 2019-11-22 11:53:08 |
Timo Aaltonen |
tags |
server-next |
server-next verification-needed verification-needed-bionic |
|
| 2019-11-28 15:21:08 |
Simon Déziel |
tags |
server-next verification-needed verification-needed-bionic |
server-next verification-done verification-done-bionic |
|
| 2019-12-02 11:15:59 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-12-02 11:26:02 |
Launchpad Janitor |
haproxy (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
