haproxy ceilometer backends missing when no public-address is found

[Shane Peters]

When using eilometer in an HA configuration with two VIPs (public private) only the local units public IP is being rendered as an available back-end in haproxy.

Relevant Config:
=================
"openstack-origin": "cloud:trusty-liberty"
"os-admin-network": 192.168.1.0/22
"os-internal-network": 192.168.1.0/22
"os-public-network": 1.2.1.0/24
vip: "192.168.1.208 1.2.1.247"
ceilometer-hacluster:
corosync_transport: unicast

Snippet from haproxy.cfg
======================
backend ceilometer_api_1.2.1.13
balance leastconn
server ceilometer-3 1.2.1.13:8767 check

backend ceilometer_api_192.168.1.47
balance leastconn
server ceilometer-6 192.168.1.134:8767 check
server ceilometer-4 192.168.1.45:8767 check
server ceilometer-3 192.168.1.47:8767 check

Note the missing public backends for 'ceilometer-4' and 'ceilometer-6'

Since connections aren't being balanced, the single (by default) ceilometer api-worker gets overwhelmed. Raising the api-worker value to match that of the cpu count is confirmed to help smooth things out.

[Jorge Niedbalski (niedbalski)]

Hello,

I can confirm that this bug is reproducible. I configured a mitaka cloud with
the latest stable release of the charms.

I configured 3 units of ceilometer relating it to hacluster, with the following settings:

juju set ceilometer vip="10.5.0.200 10.7.0.190"

juju set ceilometer os-internal-network="10.5.0.0/16"
juju set ceilometer os-public-network="10.7.0.0/24"

The resulting haproxy configuration lacks on the backend for the configured os-public-network:

root@juju-niedbalski-xenial-machine-23:/home/ubuntu# more /etc/haproxy/haproxy.cfg
global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0

defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 5000
    timeout connect 5000
    timeout client 30000
    timeout server 30000

listen stats
    bind 127.0.0.1:8888
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth admin:djT4pZm9L5WNPzTbZq6ddz5fVdz4Z2jj

frontend tcp-in_ceilometer_api
    bind *:8777
    acl net_10.7.0.100 dst 10.7.0.100/255.255.255.0
    use_backend ceilometer_api_10.7.0.100 if net_10.7.0.100
    acl net_10.5.0.96 dst 10.5.0.96/255.255.0.0
    use_backend ceilometer_api_10.5.0.96 if net_10.5.0.96
    default_backend ceilometer_api_10.5.0.96

backend ceilometer_api_10.7.0.100
    balance leastconn
    server ceilometer-1 10.7.0.100:8767 check

backend ceilometer_api_10.5.0.96
    balance leastconn
    server ceilometer-2 10.5.0.97:8767 check
    server ceilometer-0 10.5.0.73:8767 check
    server ceilometer-1 10.5.0.96:8767 check

The reason seems to be related to the lack of public-address on the cluster relation:

ubuntu@niedbalski-xenial-bastion:~/openstack-charm-testing$ juju run --unit ceilometer/1 "relation-get -r cluster:0 - ceilometer/0"
private-address: 10.5.0.73
ubuntu@niedbalski-xenial-bastion:~/openstack-charm-testing$ juju run --unit ceilometer/1 "relation-get -r cluster:0 - ceilometer/1"
private-address: 10.5.0.96
ubuntu@niedbalski-xenial-bastion:~/openstack-charm-testing$ juju run --unit ceilometer/1 "relation-get -r cluster:0 - ceilometer/2"
private-address: 10.5.0.97

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/443749
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=f36d6e5cb1ada0cf104dcd891ebcb01b075514b2
Submitter: Jenkins
Branch: master

commit f36d6e5cb1ada0cf104dcd891ebcb01b075514b2
Author: Shane Peters <email address hidden>
Date: Thu Mar 9 11:29:56 2017 -0500

    Set os-network relation data on cluster join

    By providing openstack network relation data during cluster join we
    allow haproxy backends to be configured in the corresponding os networks
    should a 'public-address' not be available.

    Change-Id: Ie370552410fffb859d236a011844427d8b3aff45
    Closes-Bug: #1669044

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/446745

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/446745
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=4eeaca33bf31f56d8b474760e23cc30c8a2012a3
Submitter: Jenkins
Branch: stable/17.02

commit 4eeaca33bf31f56d8b474760e23cc30c8a2012a3
Author: Shane Peters <email address hidden>
Date: Thu Mar 9 11:29:56 2017 -0500

    Set os-network relation data on cluster join

    By providing openstack network relation data during cluster join we
    allow haproxy backends to be configured in the corresponding os networks
    should a 'public-address' not be available.

    Change-Id: Ie370552410fffb859d236a011844427d8b3aff45
    Closes-Bug: #1669044
    (cherry picked from commit f36d6e5cb1ada0cf104dcd891ebcb01b075514b2)

[Edward Hope-Morley (hopem)]

ftr this was previously raised and fixed for ceph-radosgw in https://bugs.launchpad.net/charms/+source/ceph-radosgw/+bug/1641870