HAProxyContext on Ubuntu 14.04 generates config that fails to start on boot

Bug #1755061 reported by Paul Collins on 2018-03-12
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
haproxy (Ubuntu)
High
Unassigned
Trusty
High
James Page

Bug Description

[Impact]
Valid haproxy configuration directives don't work on trusty as /run/haproxy does not survive reboots and is not-recreated on daemon start.

[Test Case]
sudo apt install haproxy
configure /etc/haproxy.cfg with a admin socket in /run/haproxy:

global
    log /var/lib/haproxy/dev/log local0
    log /var/lib/haproxy/dev/log local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0
    stats socket /var/run/haproxy/admin.sock mode 600 level admin
    stats timeout 2m

Restart haproxy (will fail as /{,var/}run/haproxy does not exist)

[Regression Potential]
Minimal - same fix is in later package revisions

[Original Bug Report]
While testing upgrades of an Ubuntu 14.04 deployment of OpenStack from ~15.04 to 17.11 charms, I noticed that a number of the OpenStack charmed services failed to start haproxy when I rebooted their units: cinder, glance, keystone, neutron-api, nova-cloud-controller, and swift-proxy.

The following was in /var/log/boot.log:

[ALERT] 069/225906 (1100) : cannot bind socket for UNIX listener (/var/run/haproxy/admin.sock). Aborting.
[ALERT] 069/225906 (1100) : [/usr/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
 * Starting haproxy haproxy [fail]

The charm created /var/run/haproxy, but since /var/run (really /run) is a tmpfs, this did not survive the reboot and so haproxy could not create the socket.

I compared the haproxy.cfg the charm creates with the default config shipped by the Ubuntu 16.04 haproxy package, and it seems that charmhelpers/contrib/openstack/templates/haproxy.cfg is closely based on the package, including the admin.sock directive. However, on Ubuntu 16.04, /etc/init.d/haproxy ensures that /var/run/haproxy exists before it starts haproxy:

[agnew(work)] diff -u haproxy-1.4.24/debian/haproxy.init haproxy-1.6.3/debian/haproxy.init
--- haproxy-1.4.24/debian/haproxy.init 2015-12-16 03:55:29.000000000 +1300
+++ haproxy-1.6.3/debian/haproxy.init 2015-12-31 20:10:38.000000000 +1300
[...]
@@ -50,6 +41,10 @@

 haproxy_start()
 {
+ [ -d "$RUNDIR" ] || mkdir "$RUNDIR"
+ chown haproxy:haproxy "$RUNDIR"
+ chmod 2775 "$RUNDIR"
+
        check_haproxy_config

        start-stop-daemon --quiet --oknodo --start --pidfile "$PIDFILE" \
[...]

charm-helpers or the OpenStack charms or both should be updated so that haproxy will start on boot when running on Ubuntu 14.04.

Paul Collins (pjdc) wrote :

18.02 on Ubuntu 14.04 is also affected.

James Page (james-page) wrote :

The linked PR looks reasonable - however we could just SRU a fix into the haproxy package in trusty to ensure the RUNDIR is created on boot as an alternative approach?

Changed in charm-helpers:
status: New → Triaged
Changed in charm-cinder:
status: New → Triaged
Changed in charm-glance:
status: New → Triaged
Changed in charm-keystone:
status: New → Triaged
Changed in charm-neutron-api:
status: New → Triaged
Changed in charm-nova-cloud-controller:
status: New → Triaged
Changed in charm-swift-proxy:
status: New → Triaged
Changed in charm-helpers:
importance: Undecided → High
Changed in charm-cinder:
importance: Undecided → High
Changed in charm-glance:
importance: Undecided → High
Changed in charm-keystone:
importance: Undecided → High
Changed in charm-neutron-api:
importance: Undecided → High
Changed in charm-nova-cloud-controller:
importance: Undecided → High
Changed in charm-swift-proxy:
importance: Undecided → High
Paul Collins (pjdc) wrote :

That would also work -- and in fact the workaround I deployed for the upgrade was to just update /etc/init.d/haproxy to do so.

James Page (james-page) on 2018-05-18
Changed in haproxy (Ubuntu):
status: New → Triaged
importance: Undecided → High
Changed in charm-swift-proxy:
status: Triaged → Invalid
Changed in charm-nova-cloud-controller:
status: Triaged → Invalid
Changed in charm-neutron-api:
status: Triaged → Invalid
Changed in charm-keystone:
status: Triaged → Invalid
Changed in charm-glance:
status: Triaged → Invalid
Changed in charm-cinder:
status: Triaged → Invalid
Changed in charm-helpers:
status: Triaged → Invalid
James Page (james-page) wrote :

Update for trusty uploaded to SRU queue for ubuntu-sru review.

Changed in haproxy (Ubuntu):
status: Triaged → Fix Released
Changed in haproxy (Ubuntu Trusty):
importance: Undecided → High
status: New → In Progress
assignee: nobody → James Page (james-page)
description: updated
James Page (james-page) wrote :

Note that the UCA for trusty-mitaka does include a later haproxy version with the same fixes - however nice to get the stock trusty version fixed up as well!

Hello Paul, or anyone else affected,

Accepted haproxy into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.4.24-2ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-trusty
affects: charm-helpers → ubuntu-translations
no longer affects: ubuntu-translations
affects: charm-swift-proxy → ubuntu-translations
no longer affects: ubuntu-translations
affects: charm-neutron-api → ubuntu-translations
no longer affects: ubuntu-translations
affects: charm-keystone → ubuntu-translations
no longer affects: ubuntu-translations
affects: charm-cinder → ubuntu-translations
no longer affects: ubuntu-translations
affects: charm-glance → ubuntu-translations
no longer affects: ubuntu-translations
affects: charm-nova-cloud-controller → ubuntu-translations
no longer affects: ubuntu-translations

The fix for this bug has been awaiting testing feedback in the -proposed repository for trusty for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers