HAProxyContext on Ubuntu 14.04 generates config that fails to start on boot

Bug #1755061 reported by Paul Collins on 2018-03-12
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Charm Helpers
High
Unassigned
OpenStack cinder charm
High
Unassigned
OpenStack glance charm
High
Unassigned
OpenStack keystone charm
High
Unassigned
OpenStack neutron-api charm
High
Unassigned
OpenStack nova-cloud-controller charm
High
Unassigned
OpenStack swift-proxy charm
High
Unassigned
haproxy (Ubuntu)
High
Unassigned
Trusty
High
James Page

Bug Description

[Impact]
Valid haproxy configuration directives don't work on trusty as /run/haproxy does not survive reboots and is not-recreated on daemon start.

[Test Case]
sudo apt install haproxy
configure /etc/haproxy.cfg with a admin socket in /run/haproxy:

global
    log /var/lib/haproxy/dev/log local0
    log /var/lib/haproxy/dev/log local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0
    stats socket /var/run/haproxy/admin.sock mode 600 level admin
    stats timeout 2m

Restart haproxy (will fail as /{,var/}run/haproxy does not exist)

[Regression Potential]
Minimal - same fix is in later package revisions

[Original Bug Report]
While testing upgrades of an Ubuntu 14.04 deployment of OpenStack from ~15.04 to 17.11 charms, I noticed that a number of the OpenStack charmed services failed to start haproxy when I rebooted their units: cinder, glance, keystone, neutron-api, nova-cloud-controller, and swift-proxy.

The following was in /var/log/boot.log:

[ALERT] 069/225906 (1100) : cannot bind socket for UNIX listener (/var/run/haproxy/admin.sock). Aborting.
[ALERT] 069/225906 (1100) : [/usr/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
 * Starting haproxy haproxy [fail]

The charm created /var/run/haproxy, but since /var/run (really /run) is a tmpfs, this did not survive the reboot and so haproxy could not create the socket.

I compared the haproxy.cfg the charm creates with the default config shipped by the Ubuntu 16.04 haproxy package, and it seems that charmhelpers/contrib/openstack/templates/haproxy.cfg is closely based on the package, including the admin.sock directive. However, on Ubuntu 16.04, /etc/init.d/haproxy ensures that /var/run/haproxy exists before it starts haproxy:

[agnew(work)] diff -u haproxy-1.4.24/debian/haproxy.init haproxy-1.6.3/debian/haproxy.init
--- haproxy-1.4.24/debian/haproxy.init 2015-12-16 03:55:29.000000000 +1300
+++ haproxy-1.6.3/debian/haproxy.init 2015-12-31 20:10:38.000000000 +1300
[...]
@@ -50,6 +41,10 @@

 haproxy_start()
 {
+ [ -d "$RUNDIR" ] || mkdir "$RUNDIR"
+ chown haproxy:haproxy "$RUNDIR"
+ chmod 2775 "$RUNDIR"
+
        check_haproxy_config

        start-stop-daemon --quiet --oknodo --start --pidfile "$PIDFILE" \
[...]

charm-helpers or the OpenStack charms or both should be updated so that haproxy will start on boot when running on Ubuntu 14.04.

Paul Collins (pjdc) wrote :

18.02 on Ubuntu 14.04 is also affected.

James Page (james-page) wrote :

The linked PR looks reasonable - however we could just SRU a fix into the haproxy package in trusty to ensure the RUNDIR is created on boot as an alternative approach?

Changed in charm-helpers:
status: New → Triaged
Changed in charm-cinder:
status: New → Triaged
Changed in charm-glance:
status: New → Triaged
Changed in charm-keystone:
status: New → Triaged
Changed in charm-neutron-api:
status: New → Triaged
Changed in charm-nova-cloud-controller:
status: New → Triaged
Changed in charm-swift-proxy:
status: New → Triaged
Changed in charm-helpers:
importance: Undecided → High
Changed in charm-cinder:
importance: Undecided → High
Changed in charm-glance:
importance: Undecided → High
Changed in charm-keystone:
importance: Undecided → High
Changed in charm-neutron-api:
importance: Undecided → High
Changed in charm-nova-cloud-controller:
importance: Undecided → High
Changed in charm-swift-proxy:
importance: Undecided → High
Paul Collins (pjdc) wrote :

That would also work -- and in fact the workaround I deployed for the upgrade was to just update /etc/init.d/haproxy to do so.

James Page (james-page) on 2018-05-18
Changed in haproxy (Ubuntu):
status: New → Triaged
importance: Undecided → High
Changed in charm-swift-proxy:
status: Triaged → Invalid
Changed in charm-nova-cloud-controller:
status: Triaged → Invalid
Changed in charm-neutron-api:
status: Triaged → Invalid
Changed in charm-keystone:
status: Triaged → Invalid
Changed in charm-glance:
status: Triaged → Invalid
Changed in charm-cinder:
status: Triaged → Invalid
Changed in charm-helpers:
status: Triaged → Invalid
James Page (james-page) wrote :

Update for trusty uploaded to SRU queue for ubuntu-sru review.

Changed in haproxy (Ubuntu):
status: Triaged → Fix Released
Changed in haproxy (Ubuntu Trusty):
importance: Undecided → High
status: New → In Progress
assignee: nobody → James Page (james-page)
description: updated
James Page (james-page) wrote :

Note that the UCA for trusty-mitaka does include a later haproxy version with the same fixes - however nice to get the stock trusty version fixed up as well!

Hello Paul, or anyone else affected,

Accepted haproxy into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.4.24-2ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-trusty
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers