haproxy fails on arm64 due to alignment error

Bug #1804069 reported by Jonathan Rosser
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
haproxy (Ubuntu)
Fix Released
High
Unassigned
Bionic
Fix Released
High
Andreas Hasenack
Cosmic
Fix Released
High
Andreas Hasenack

Bug Description

[Impact]
haproxy as shipped with bionic and cosmic doesn't work on arm64 architectures, crashing the moment it serves a request.

[Test Case]

* install haproxy and apache in an up-to-date ubuntu release you are testing, in an arm64 system:
sudo apt update && sudo apt dist-upgrade -y && sudo apt install haproxy apache2 -y

* Create /etc/haproxy/haproxy.cfg with the following contents:
global
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        maxconn 4096

defaults
        log global
        option dontlognull
        option redispatch
        retries 3
        timeout client 50s
        timeout connect 10s
        timeout http-request 5s
        timeout server 50s
        maxconn 4096

frontend test-front
    bind *:8080
    mode http
    default_backend test-back

backend test-back
    mode http
    stick store-request src
    stick-table type ip size 256k expire 30m
    server test-1 localhost:80

* in one terminal, keep tailing the (still nonexistent) haproxy log file:
tail -F /var/log/haproxy.log

* in another terminal, restart haproxy:
sudo systemctl restart haproxy

* The haproxy log will become live, and already show errors:
Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : Exiting Master process...
Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [ALERT] 023/191958 (2286) : Current worker 2287 exited with code 143
Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : All workers exited. Exiting... (143)

* Run wget to try to fetch the apache frontpage, via haproxy, limited to one attempt. It will fail:
$ wget -t1 http://localhost:8080
--2019-01-24 19:23:51-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... No data received.
Giving up.
$ echo $?
4

* the haproxy logs will show errors:
Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : Current worker 6412 exited with code 135
Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : exit-on-failure: killing every workers with SIGTERM
Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [WARNING] 023/192351 (6411) : All workers exited. Exiting... (135)

* Update the haproxy package and try the wget one more time. This time it will work, and the haproxy logs will stay silent:

$ wget -t1 http://localhost:8080
--2019-01-24 19:26:14-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10918 (11K) [text/html]
Saving to: ‘index.html’

index.html 100%[================================================================================================================================>] 10.66K --.-KB/s in 0s

2019-01-24 19:26:14 (75.3 MB/s) - ‘index.html’ saved [10918/10918]

[Regression Potential]
Patch was applied upstream in 1.8.15 and is available in the same form in the latest 1.8.17 release. The patch is a bit low level, but seems to have been well understood.

[Other Info]
After writing the testing instructions for this bug, I decided they could be easily converted to a DEP8 test, which I did and included in this SRU. This new test, very simple but effective, shows that arm64 is working, and that the other architectures didn't break.

[Original Description]

This fault was reported via the haproxy mailing list https://<email address hidden>/msg31749.html

And then patched in the haproxy code here https://github.com/haproxy/haproxy/commit/52dabbc4fad338233c7f0c96f977a43f8f81452a

Without this patch haproxy is not functional on aarch64/arm64. Experimental deployments of openstack-ansible on arm64 fail because of this bug, and without a fix applied to the ubuntu bionic packages we cannot proceed further as the openstack CI only installs the upstream ubuntu distribution packages.

Related branches

Revision history for this message
Karl Stenerud (kstenerud) wrote :

Hi Jonathan,

Thanks for reporting this bug! Can you put together a simple test case that reproduces the bug so that I can verify bug and fix?

Changed in haproxy (Ubuntu):
status: New → Incomplete
Revision history for this message
Jonathan Rosser (jrosser) wrote :

Hi Karl,

I've asked the reporter of the original haproxy bug to help out with a test case - I don't have access to an arm64 system.

Revision history for this message
Paul Martin (paul-martin-1) wrote :

Minimal config that duplicates this:

--

global
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        maxconn 4096

defaults
        log global
        option dontlognull
        option redispatch
        retries 3
        timeout client 50s
        timeout connect 10s
        timeout http-request 5s
        timeout server 50s
        maxconn 4096

frontend test-front
    bind *:80
    mode http
    default_backend test-back

backend test-back
    mode http
    stick store-request src
    stick-table type ip size 256k expire 30m
    server www.ubuntu.com 91.189.89.115:80

--

Then:

# curl http://127.0.0.1/
curl: (52) Empty reply from server

and you see in /var/log/haproxy.log:

Nov 20 13:52:11 localhost haproxy[43712]: [ALERT] 323/135209 (43712) : Current worker 43713 exited with code 135
Nov 20 13:52:11 localhost haproxy[43712]: [ALERT] 323/135209 (43712) : exit-on-failure: killing every workers with SIGTERM
Nov 20 13:52:11 localhost haproxy[43712]: [WARNING] 323/135209 (43712) : All workers exited. Exiting... (135)

Changed in haproxy (Ubuntu):
status: Incomplete → Triaged
importance: Undecided → High
Robie Basak (racb)
tags: added: bitesize server-next
Changed in haproxy (Ubuntu Bionic):
status: New → Triaged
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

This was fixed upstream in 1.8.15:
2018/12/13 : 1.8.15
    - MINOR: threads: Make sure threads_sync_pipe is initialized before using it.
...
    - BUG/MEDIUM: Make sure stksess is properly aligned. <------

Marking main task as fix released, as disco has 1.8.17. But confirmed for bionic and cosmic (assuming it was introduced in the 1.8.x series and not present earlier).

Changed in haproxy (Ubuntu):
status: Triaged → Fix Released
Changed in haproxy (Ubuntu Cosmic):
status: New → Triaged
Changed in haproxy (Ubuntu Bionic):
importance: Undecided → High
Changed in haproxy (Ubuntu Cosmic):
importance: Undecided → High
Changed in haproxy (Ubuntu Bionic):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in haproxy (Ubuntu Cosmic):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in haproxy (Ubuntu Bionic):
status: Triaged → In Progress
Changed in haproxy (Ubuntu Cosmic):
status: Triaged → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Bug reproduced, and fix confirmed.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Tests kicked off:
cosmic: https://bileto.ubuntu.com/#/ticket/3610
bionic: https://bileto.ubuntu.com/#/ticket/3611

Those tickets have links to respective ppa builds, if someone wants to test early.

description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Jonathan, or anyone else affected,

Accepted haproxy into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.8.13-2ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in haproxy (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in haproxy (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Jonathan, or anyone else affected,

Accepted haproxy into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.8.8-1ubuntu0.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Jonathan, could you please detail what exactly you tested, and with which package version? The SRU team will appreciate that level of detail, which gives more confidence in the update.

Revision history for this message
Paul Martin (paul-martin-1) wrote :

I repeated the above tests on Bionic on a Qualcomm Centriq processor with the package http://launchpadlibrarian.net/409244670/haproxy_1.8.8-1ubuntu0.4_arm64.deb

Revision history for this message
Jonathan Rosser (jrosser) wrote :

Andreas, Paul checked the bionic package so comment #10 should hopefully be sufficient. I am currently unable to boot the cosmic cloud aarch64 cloud image so verifying that might take some more time.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

You can boot some other release for the host and use a lxd container for the cosmic test.

Revision history for this message
Jonathan Rosser (jrosser) wrote :

Andreas,

As far as cosmic goes

* I have no actual ARM hardware
* The cosmic cloud image will not boot on the cloud provider I have an account with
* I can boot the aarch64 bionic cloud image
* LXD will install
* Creating a cosmic LXD container fails because systemd-networkd will not start within the container, therefore there is no networking.

Feb 06 10:28:26 cosmic systemd[435]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
Feb 06 10:28:26 cosmic systemd[435]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied

At this point i'm a bit stuck.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Ok, thanks for trying. I can complete the verification and will post results later today.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Cosmic verification

Using arm64:
root@cosmic-haproxy-1804069:~# uname -a
Linux cosmic-haproxy-1804069 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:32:18 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

First, confirming the bug:
root@cosmic-haproxy-1804069:~# apt-cache policy haproxy
haproxy:
  Installed: 1.8.13-2ubuntu0.1
  Candidate: 1.8.13-2ubuntu0.1
  Version table:
 *** 1.8.13-2ubuntu0.1 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-updates/main arm64 Packages

wget fails as expected:
root@cosmic-haproxy-1804069:~# wget -t1 http://localhost:8080
--2019-02-06 11:43:52-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... No data received.
Giving up.

root@cosmic-haproxy-1804069:~# echo $?
4

Haproxy error logs show:
Feb 6 11:43:52 cosmic-haproxy-1804069 haproxy[4539]: [ALERT] 036/114339 (4539) : Current worker 4540 exited with code 135
Feb 6 11:43:52 cosmic-haproxy-1804069 haproxy[4539]: [ALERT] 036/114339 (4539) : exit-on-failure: killing every workers with SIGTERM
Feb 6 11:43:52 cosmic-haproxy-1804069 haproxy[4539]: [WARNING] 036/114339 (4539) : All workers exited. Exiting... (135)

Upgraded haproxy package:
root@cosmic-haproxy-1804069:~# apt-cache policy haproxy
haproxy:
  Installed: 1.8.13-2ubuntu0.2
  Candidate: 1.8.13-2ubuntu0.2
  Version table:
 *** 1.8.13-2ubuntu0.2 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-proposed/main arm64 Packages

wget now works:
root@cosmic-haproxy-1804069:~# wget -t1 http://localhost:8080
--2019-02-06 11:46:19-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10918 (11K) [text/html]
Saving to: ‘index.html’

index.html 100%[================================================================================================================================>] 10.66K --.-KB/s in 0s

2019-02-06 11:46:19 (154 MB/s) - ‘index.html’ saved [10918/10918]

root@cosmic-haproxy-1804069:~# echo $?
0
root@cosmic-haproxy-1804069:~#

And the haproxy logs stay quiet.

Cosmic verification succeeded.

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Bionic verification

arm system:
root@bionic-haproxy-1804069:~# uname -a
Linux bionic-haproxy-1804069 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:32:18 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

using affected package at first:
root@bionic-haproxy-1804069:~# apt-cache policy haproxy
haproxy:
  Installed: 1.8.8-1ubuntu0.3
  Candidate: 1.8.8-1ubuntu0.3
  Version table:
 *** 1.8.8-1ubuntu0.3 500
        500 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 Packages

wget fails as expected:
root@bionic-haproxy-1804069:~# wget -t1 http://localhost:8080
--2019-02-06 11:57:56-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... No data received.
Giving up.

root@bionic-haproxy-1804069:~# echo $?
4

haproxy logs show errors:
Feb 6 11:57:57 bionic-haproxy-1804069 haproxy[10191]: [ALERT] 036/115744 (10191) : Current worker 10192 exited with code 135
Feb 6 11:57:57 bionic-haproxy-1804069 haproxy[10191]: [ALERT] 036/115744 (10191) : exit-on-failure: killing every workers with SIGTERM
Feb 6 11:57:57 bionic-haproxy-1804069 haproxy[10191]: [WARNING] 036/115744 (10191) : All workers exited. Exiting... (135)

Using updated package:
root@bionic-haproxy-1804069:~# apt-cache policy haproxy
haproxy:
  Installed: 1.8.8-1ubuntu0.4
  Candidate: 1.8.8-1ubuntu0.4
  Version table:
 *** 1.8.8-1ubuntu0.4 500
        500 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main arm64 Packages

wget now works:
root@bionic-haproxy-1804069:~# wget -t1 http://localhost:8080
--2019-02-06 12:00:17-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10918 (11K) [text/html]
Saving to: ‘index.html’

index.html 100%[================================================================================================================================>] 10.66K --.-KB/s in 0s

2019-02-06 12:00:17 (165 MB/s) - ‘index.html’ saved [10918/10918]

root@bionic-haproxy-1804069:~# echo $?
0

and haproxy logs stay silent.

bionic verification succeeded

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for haproxy has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.8.13-2ubuntu0.2

---------------
haproxy (1.8.13-2ubuntu0.2) cosmic; urgency=medium

  * d/p/stksess-align.patch: Make sure stksess is properly aligned.
    (LP: #1804069)
  * d/t/control, d/t/proxy-localhost: simple DEP8 test to actually
    generate traffic through haproxy.

 -- Andreas Hasenack <email address hidden> Wed, 23 Jan 2019 17:24:30 -0200

Changed in haproxy (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.8.8-1ubuntu0.4

---------------
haproxy (1.8.8-1ubuntu0.4) bionic; urgency=medium

  * d/p/stksess-align.patch: Make sure stksess is properly aligned.
    (LP: #1804069)
  * d/t/control, d/t/proxy-localhost: simple DEP8 test to actually
    generate traffic through haproxy.

 -- Andreas Hasenack <email address hidden> Thu, 24 Jan 2019 10:20:49 -0200

Changed in haproxy (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.