HAProxyContext on Ubuntu 14.04 generates config that fails to start on boot

[Paul Collins]

[Impact]
Valid haproxy configuration directives don't work on trusty as /run/haproxy does not survive reboots and is not-recreated on daemon start.

[Test Case]
sudo apt install haproxy
configure /etc/haproxy.cfg with a admin socket in /run/haproxy:

global
    log /var/lib/haproxy/dev/log local0
    log /var/lib/haproxy/dev/log local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0
    stats socket /var/run/haproxy/admin.sock mode 600 level admin
    stats timeout 2m

Restart haproxy (will fail as /{,var/}run/haproxy does not exist)

[Regression Potential]
Minimal - same fix is in later package revisions

[Original Bug Report]
While testing upgrades of an Ubuntu 14.04 deployment of OpenStack from ~15.04 to 17.11 charms, I noticed that a number of the OpenStack charmed services failed to start haproxy when I rebooted their units: cinder, glance, keystone, neutron-api, nova-cloud-controller, and swift-proxy.

The following was in /var/log/boot.log:

[ALERT] 069/225906 (1100) : cannot bind socket for UNIX listener (/var/run/haproxy/admin.sock). Aborting.
[ALERT] 069/225906 (1100) : [/usr/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
 * Starting haproxy haproxy [fail]

The charm created /var/run/haproxy, but since /var/run (really /run) is a tmpfs, this did not survive the reboot and so haproxy could not create the socket.

I compared the haproxy.cfg the charm creates with the default config shipped by the Ubuntu 16.04 haproxy package, and it seems that charmhelpers/contrib/openstack/templates/haproxy.cfg is closely based on the package, including the admin.sock directive. However, on Ubuntu 16.04, /etc/init.d/haproxy ensures that /var/run/haproxy exists before it starts haproxy:

[agnew(work)] diff -u haproxy-1.4.24/debian/haproxy.init haproxy-1.6.3/debian/haproxy.init
--- haproxy-1.4.24/debian/haproxy.init 2015-12-16 03:55:29.000000000 +1300
+++ haproxy-1.6.3/debian/haproxy.init 2015-12-31 20:10:38.000000000 +1300
[...]
@@ -50,6 +41,10 @@

 haproxy_start()
 {
+ [ -d "$RUNDIR" ] || mkdir "$RUNDIR"
+ chown haproxy:haproxy "$RUNDIR"
+ chmod 2775 "$RUNDIR"
+
        check_haproxy_config

        start-stop-daemon --quiet --oknodo --start --pidfile "$PIDFILE" \
[...]

charm-helpers or the OpenStack charms or both should be updated so that haproxy will start on boot when running on Ubuntu 14.04.

[Paul Collins (pjdc)]

18.02 on Ubuntu 14.04 is also affected.

[Paul Collins (pjdc)]

PR: https://github.com/juju/charm-helpers/pull/130

[James Page (james-page)]

The linked PR looks reasonable - however we could just SRU a fix into the haproxy package in trusty to ensure the RUNDIR is created on boot as an alternative approach?

[Paul Collins (pjdc)]

That would also work -- and in fact the workaround I deployed for the upgrade was to just update /etc/init.d/haproxy to do so.

[James Page (james-page)]

Update for trusty uploaded to SRU queue for ubuntu-sru review.

[James Page (james-page)]

Note that the UCA for trusty-mitaka does include a later haproxy version with the same fixes - however nice to get the stock trusty version fixed up as well!

[Brian Murray (brian-murray)]

Hello Paul, or anyone else affected,

Accepted haproxy into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.4.24-2ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for trusty for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Łukasz Zemczak (sil2100)]

The version of haproxy in the proposed pocket of Trusty that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.