HAproxy binds to IPv6 even if it's disabled in the OS and as a result fails to start

[Tytus Kurek]

This issue has been faced when deploying designate charm from openstack-charmers-next repository in the HA mode on Ubuntu Trusty where IPv6 is disabled:

$ juju status designate
Model Controller Cloud/Region Version SLA
default bratislava bratislava 2.2.2 unsupported

App Version Status Scale Charm Store Rev OS Notes
designate 2.0.0 blocked 3 designate local 33 ubuntu
designate-hacluster active 3 hacluster local 31 ubuntu

Unit Workload Agent Machine Public address Ports Message
designate/51* blocked idle 22/lxd/23 100.107.2.1 9001/tcp Ports which should be open, but are not: 9001
  designate-hacluster/96 active executing 100.107.2.1 Unit is ready and clustered
designate/53 blocked idle 23/lxd/23 100.107.66.239 9001/tcp Ports which should be open, but are not: 9001
  designate-hacluster/97* active executing 100.107.66.239 Unit is ready and clustered
designate/54 blocked idle 24/lxd/23 100.107.66.242 9001/tcp Ports which should be open, but are not: 9001
  designate-hacluster/95 active executing 100.107.66.242 Unit is ready and clustered

Machine State DNS Inst id Series AZ Message
22 started 100.107.2.134 q8eh4f trusty FC1 Deployed
22/lxd/23 started 100.107.2.1 juju-64cbea-22-lxd-23 trusty Container started
23 started 100.107.66.126 fbg4ta trusty FC2 Deployed
23/lxd/23 started 100.107.66.239 juju-64cbea-23-lxd-23 trusty Container started
24 started 100.107.66.128 gdpgys trusty FC2 Deployed
24/lxd/23 started 100.107.66.242 juju-64cbea-24-lxd-23 trusty Container started

Relation Provides Consumes Type
cluster designate designate peer
dns-backend designate designate-bind regular
ha designate designate-hacluster subordinate
identity-service designate keystone regular
shared-db designate percona-cluster regular
amqp designate rabbitmq-server-designate regular
hanode designate-hacluster designate-hacluster peer

This is because "/etc/haproxy/haproxy.cfg" file is being written with the following content:

global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0

defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 5000
    timeout connect 5000
    timeout client 30000
    timeout server 30000

listen stats
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth admin:oetycPlNCmEMfFKN2DfcdV0OfPPPliH1

frontend tcp-in_designate-api_admin
    bind *:9001
    bind :::9001
    acl net_100.107.2.1 dst 100.107.2.1/255.255.252.0
    use_backend designate-api_admin_100.107.2.1 if net_100.107.2.1
    default_backend designate-api_admin_100.107.2.1

backend designate-api_admin_100.107.2.1
    balance leastconn
    server designate-51 100.107.2.1:8991 check
    server designate-53 100.107.66.239:8991 check
    server designate-54 100.107.66.242:8991 check

The problem is that "bind :::9001" statement won't work on the systems with IPv6 disabled and thus, will result in haproxy failing to start. It looks like the base openstack layer does not contain a check for this statement in "haproxy.cfg" template file.

[Tytus Kurek (tkurek)]

Fix proposed to branch: master
Review: https://review.openstack.org/507033/

[James Page (james-page)]

Looking at the main charmhelpers template, things work exactly the same way there as well.

[James Page (james-page)]

frontend tcp-in_{{ service }}
    bind *:{{ ports[0] }}
    bind :::{{ ports[0] }}
    {% for frontend in frontends -%}
    acl net_{{ frontend }} dst {{ frontends[frontend]['network'] }}
    use_backend {{ service }}_{{ frontend }} if net_{{ frontend }}
    {% endfor -%}
    default_backend {{ service }}_{{ default_backend }}

[James Page (james-page)]

Tytus - do you see this issue with non-reactive charms as well?

[Tytus Kurek (tkurek)]

It looks like templates for other charms have been modified manually in this environment:

frontend tcp-in_{{ service }}
    bind *:{{ ports[0] }}
    {% if ipv6 -%}
    bind :::{{ ports[0] }}
    {% endif -%}

It looks like it's a general issue.

[Tytus Kurek (tkurek)]

It is exposed in file "charmhelpers/contrib/openstack/context.py" file:

        if config('prefer-ipv6'):
            ctxt['ipv6'] = True
            ctxt['local_host'] = 'ip6-localhost'
            ctxt['haproxy_host'] = '::'

[James Page (james-page)]

That approach is not 100% reliable as its possible that network spaces or config provided networks might also be IPv6 subnet configurations so I'm reticent to tie this to the prefer-ipv6 feature (which I'd really like to drop from the charms as it was implemented to support IPv6 deployments in the context of a very old juju version).

[James Page (james-page)]

It also breaks the dual-stack work done last cycle, which depends on this binding so I'm a little worried there are alot of moving parts in play here that might break.

[James Page (james-page)]

We could conditionally enable the IPv6 binding IF IPv6 is enabled on the host system - that would probably work in all use-cases.

[James Page (james-page)]

The ip charmhelper has a is_ipv6_disabled() function - that information could be passed via a context to the templates so that this sort of thing can be conditionally enabled.

[James Page (james-page)]

Raising charm-helpers target as well - we'll need to make this part of the HAProxy context and template for classic charm support for running with IPv6 disabled.

[Tytus Kurek (tkurek)]

Fix for Charm Helpers proposed to the following branch:

lp:~tkurek/charm-helpers/1719280

[James Page (james-page)]

Hi Tytus

Charm Helpers development moved about two weeks ago over to github - please can you raise your changes as a pull requests against:

    https://github.com/juju/charm-helpers

Cheers

Jamers

[James Page (james-page)]

No - charm-helpers is not part of openstack; you'll need to fork the project on github.com to your own account, create a branch on that repository, make your changes, push back to your repo copy on github.com and then raise a pull request (which will automatically pop up in the WebUI).

[Tytus Kurek (tkurek)]

Pull request created.

[Tytus Kurek (tkurek)]

Fix for charms.openstack proposed to branch: master
Review: https://review.openstack.org/#/c/508888/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/508888
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=82ba9595fa05dc6ddccabb3242b2f9c222f74c6e
Submitter: Zuul
Branch: master

commit 82ba9595fa05dc6ddccabb3242b2f9c222f74c6e
Author: Tytus Kurek <email address hidden>
Date: Mon Oct 2 16:06:33 2017 +0200

    Enable IPv6 check in haproxy template file

    This patchset exposes "ipv6_enabled" variable which can be used to
    add IPv6 check to the haproxy template file.

    Change-Id: I1e7329323cbe4ea9d2d79e4196690efa015aa0f4
    Closes-Bug: #1719280