haproxy.cfg backends do not reflect all cluster members - no loadbalancing

[Drew Freiberger]

When configuring gnocchi with separate admin-space, internal-space, and external-space networks, and setting multiple VIPs for internal-space and external-space, but not admin-space, along with having the "cluster" binding be on the admin-space, the backends in haproxy.cfg is missing non-localserver backends (gnocchi-0 only has gnocchi-0 backend for external/internal network frontends, gnocchi-1 only has itself, etc) such as:

frontend tcp-in_gnocchi-api_admin
    bind *:8041
    bind :::8041
    acl net_192.168.1.39 dst 192.168.1.2/255.255.0.0
    use_backend gnocchi-api_admin_192.168.1.2 if net_192.168.1.2
    acl net_192.168.2.52 dst 192.168.2.2/255.255.0.0
    use_backend gnocchi-api_admin_192.168.2.2 if net_192.168.2.2
    acl net_192.168.0.82 dst 192.168.0.2/255.255.0.0
    use_backend gnocchi-api_admin_192.168.0.2 if net_192.168.0.2
    default_backend gnocchi-api_admin_192.168.0.2

backend gnocchi-api_admin_192.168.1.2
    balance leastconn
    server gnocchi-1 192.168.1.2:8031 check

backend gnocchi-api_admin_192.168.2.2
    balance leastconn
    server gnocchi-1 192.168.2.2:8031 check

backend gnocchi-api_admin_192.168.0.82
    balance leastconn
    server gnocchi-1 192.168.0.2:8031 check
    server gnocchi-2 192.168.0.3:8031 check
    server gnocchi-3 192.168.0.4:8031 check

to reproduce, create 3 networks, register 3 juju spaces, deploy gnocchi and hacluster along with basic keystone/mysql/rmq setup, configure gnocchi as such:

if your spaces are
oam-space: 192.168.0.0/24
internal-space: 192.168.1.0/24
external-space: 192.168.2.0/24

gnocchi:
  options:
    vip: 192.168.1.254 192.168.2.254
  bindings:
    "": oam-space
    cluster: oam-space
    internal: internal-space
    admin: external-space
    public: external-space

Once this setup is in place, the relation-get between gnocchi units will show only 192.168.0.X/24 addresses. When the gnocchi charm goes to configure the HAProxyContext, the cluster members for the front-ends on the external-space and internal-space do not have matching backend IPs, so this line in the template fails to render the other two backends for the functional networks, and only configures the incoming interface on the "cluster" binding interface as being load-balanced.

In the gnocchi charm, there is the file templates/haproxy.cfg
which uses this logic for adding backends to a service:
    {% for unit, address in cluster.cluster_hosts[frontend]['backends'].items() -%}

Somehow, the gnocchi charm must detect all VIP-related network IPs the communicate over the cluster relation to allow for all frontends to be properly loadbalanced.

Expected result:

backend gnocchi-api_admin_192.168.1.2
    balance leastconn
    server gnocchi-1 192.168.1.2:8031 check
    server gnocchi-1 192.168.1.3:8031 check
    server gnocchi-1 192.168.1.4:8031 check

backend gnocchi-api_admin_192.168.2.2
    balance leastconn
    server gnocchi-1 192.168.2.2:8031 check
    server gnocchi-1 192.168.2.3:8031 check
    server gnocchi-1 192.168.2.4:8031 check

backend gnocchi-api_admin_192.168.0.82
    balance leastconn
    server gnocchi-1 192.168.0.2:8031 check
    server gnocchi-2 192.168.0.3:8031 check
    server gnocchi-3 192.168.0.4:8031 check

This is interweaved with lp#1857039 which points out that the 192.168.0.0/24 network doesn't get an apache vhost because it isn't a member IP of one of the admin/internal/public bindings.

[Drew Freiberger (afreiberger)]

Subscribing field-high as this is affecting gnocchi performance on a production site and loadbalancing the API across the 3 units may help to alleviate the pressure on this service.

[Alvaro Uria (aluria)]

charm-gnocchi metadata.yaml file does not have a "hacluster" interface [1].

https://github.com/openstack/charm-gnocchi/blob/stable/20.02/src/metadata.yaml

[Alvaro Uria (aluria)]

Sorry, charm-gnocchi includes a layer called openstack-api, which does contain a "hacluster" interface [1]. After following several other layers included in other layers (openstack-principle, openstack), I also found the haproxy template [2].

1. https://github.com/openstack/charm-layer-openstack-api/blob/master/metadata.yaml
2. https://github.com/openstack/charm-layer-openstack/blob/master/templates/haproxy.cfg

[David Ames (thedac)]

TRIAGE:

I still need to get a setup to fully reproduce this. I am going to get more members of my team to look at this.

As Alvaro has pointed out the bug is in layer-openstack (and possibly charms.openstack) and thus affects all reactive charms (at least).

Still need to verify how classic charms behave in this scenario.

First a challenge to what our goal state should look like:

I would argue that the state we want is one in which *all* back end servers are in the space that cluster is bound to. We are asserting the communication between peers (on the cluster relation) should use the space cluster is bound to. So regardless of the front end IP *all* back end servers should use their cluster relation IP.

It should look something like this:

backend gnocchi-api_admin_192.168.1.2
    balance leastconn
    server gnocchi-1 192.168.0.2:8031 check
    server gnocchi-2 192.168.0.3:8031 check
    server gnocchi-3 192.168.0.4:8031 check

backend gnocchi-api_admin_192.168.2.2
    balance leastconn
    server gnocchi-1 192.168.0.2:8031 check
    server gnocchi-2 192.168.0.3:8031 check
    server gnocchi-3 192.168.0.4:8031 check

backend gnocchi-api_admin_192.168.0.82
    balance leastconn
    server gnocchi-1 192.168.0.2:8031 check
    server gnocchi-2 192.168.0.3:8031 check
    server gnocchi-3 192.168.0.4:8031 check

Next steps:

Determine the (team agreed upon) goal state
Re-create in a testing env (possibly with juju 2.8.x and openstack provider spaces)
Confirm the behavior of classic charms in a similar scenario
Determine if this is a bug in charms.openstack or layer-openstack or something else.
Fix all the bugs

[David Ames (thedac)]

After looking at LP#1857039 and testing a few things. I think I am wrong. Because of SSL and certificates and because the traffic path is haproxy -> apache -> app Drew's OP expected goal state is correct:

backend gnocchi-api_admin_192.168.1.2
    balance leastconn
    server gnocchi-1 192.168.1.2:8031 check
    server gnocchi-1 192.168.1.3:8031 check
    server gnocchi-1 192.168.1.4:8031 check

backend gnocchi-api_admin_192.168.2.2
    balance leastconn
    server gnocchi-1 192.168.2.2:8031 check
    server gnocchi-1 192.168.2.3:8031 check
    server gnocchi-1 192.168.2.4:8031 check

backend gnocchi-api_admin_192.168.0.82
    balance leastconn
    server gnocchi-1 192.168.0.2:8031 check
    server gnocchi-2 192.168.0.3:8031 check
    server gnocchi-3 192.168.0.4:8031 check

So the TRIAGE is to get haproxy to render like above.

[David Ames (thedac)]

Finally have a re-creator test.

I can confirm that classic charms conform to the expectation:

A snippet from keystone's haproxy.cfg

frontend tcp-in_public-port
    bind *:5000
    bind :::5000
    acl net_172.20.0.36 dst 172.20.0.36/255.255.255.0
    use_backend public-port_172.20.0.36 if net_172.20.0.36
    acl net_172.16.0.35 dst 172.16.0.35/255.255.255.0
    use_backend public-port_172.16.0.35 if net_172.16.0.35
    acl net_192.168.100.29 dst 192.168.100.29/255.255.255.0
    use_backend public-port_192.168.100.29 if net_192.168.100.29
    default_backend public-port_192.168.100.29

backend public-port_172.20.0.36
    balance leastconn
    server keystone-0 172.20.0.36:4990 check
    server keystone-1 172.20.0.6:4990 check
    server keystone-2 172.20.0.15:4990 check

backend public-port_172.16.0.35
    balance leastconn
    server keystone-0 172.16.0.35:4990 check
    server keystone-1 172.16.0.41:4990 check
    server keystone-2 172.16.0.21:4990 check

backend public-port_192.168.100.29
    balance leastconn
    server keystone-0 192.168.100.29:4990 check
    server keystone-1 192.168.100.21:4990 check
    server keystone-2 192.168.100.12:4990 check

Digging into the root cause now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/714792

[David Ames (thedac)]

Adding the following in the reactive/gnocchi_handlers.py file is the work around and root cause.

@reactive.when('cluster.available')
def update_peers(cluster):
    """Inform peers about this unit"""
    with charm.provide_charm_instance() as charm_instance:
        charm_instance.update_peers(cluster)

This is a proof of concept fix. My team and I will discuss if this ought to live in layer-openstack-api as this problem affects multiple reactive api charms.

https://review.opendev.org/714792

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715067

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715067
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=720a5922db534661c26c37f9bc136f206f179cde
Submitter: Zuul
Branch: master

commit 720a5922db534661c26c37f9bc136f206f179cde
Author: David Ames <email address hidden>
Date: Wed Mar 25 11:23:38 2020 -0700

    Enable update_peers method on cluster.available

    Closes-Bug: #1858132
    Change-Id: I1977e2c9625952eb22a63be0da0d6979c9f23072

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714792
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=d6622db11d400d4321739bcddc7870ee4f2879d0
Submitter: Zuul
Branch: master

commit d6622db11d400d4321739bcddc7870ee4f2879d0
Author: David Ames <email address hidden>
Date: Tue Mar 24 23:10:42 2020 +0000

    Fix HAProxy back ends

    Requires changes to charms.openstack and layer-openstack-api.

    Change-Id: Iaf20a0cd7bd2bc9accb20bd8987ed87289059e1c
    Closes-Bug: #1858132
    Depends-On: Iacfdc2a0247f35d0bddfef47c9dd5d4cf66cd309

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715558

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715560

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715563

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715564

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715565

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715567

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/715568

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715568
Committed: https://git.openstack.org/cgit/openstack/charm-placement/commit/?id=b4166e949cd9154975a1c526120c677089a204db
Submitter: Zuul
Branch: master

commit b4166e949cd9154975a1c526120c677089a204db
Author: David Ames <email address hidden>
Date: Fri Mar 27 15:02:42 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Iaa3fc3f2401981f6f0a2130a277a6da7439cff52
    Closes-Bug: #1858132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715565
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/commit/?id=ef7a12ea5460a954db2e8a78d3897494a97e8c0c
Submitter: Zuul
Branch: master

commit ef7a12ea5460a954db2e8a78d3897494a97e8c0c
Author: David Ames <email address hidden>
Date: Fri Mar 27 15:00:09 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Ibb9d67095c85f0486d24bec01c356c7e7bfb008a
    Closes-Bug: #1858132

[James Page (james-page)]

Picking up this bug as David is out this week - and it resolves a number of issues with regards to haproxy configuration.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/716535

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716565

[OpenStack Infra (hudson-openstack)]

Change abandoned by James Page (<email address hidden>) on branch: master
Review: https://review.opendev.org/716535

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715564
Committed: https://git.openstack.org/cgit/openstack/charm-masakari/commit/?id=5bee0414cf07376b85ce76c3ae4f624e52f43d10
Submitter: Zuul
Branch: master

commit 5bee0414cf07376b85ce76c3ae4f624e52f43d10
Author: David Ames <email address hidden>
Date: Fri Mar 27 14:59:22 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: I7ccc0504fd57622d08fe90eaef62f46097793780
    Closes-Bug: #1858132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715558
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=38353df4d2f1120bcbf383a2e6c537004ccc52e7
Submitter: Zuul
Branch: master

commit 38353df4d2f1120bcbf383a2e6c537004ccc52e7
Author: David Ames <email address hidden>
Date: Fri Mar 27 14:43:15 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: I1bf1f7272ab42c248ae86fede17c5cdd5f3892bc
    Closes-Bug: #1858132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715563
Committed: https://git.openstack.org/cgit/openstack/charm-manila/commit/?id=c1c2278530f2b852bbbb1f1df156225d0df5f3a2
Submitter: Zuul
Branch: master

commit c1c2278530f2b852bbbb1f1df156225d0df5f3a2
Author: David Ames <email address hidden>
Date: Fri Mar 27 14:57:18 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Ic9b25431c1f5574bad855731d0210a9dbedacfc7
    Closes-Bug: #1858132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716565
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=3b50657d13cd70e94cc0fe14efb588cc99a3f466
Submitter: Zuul
Branch: stable/20.02

commit 3b50657d13cd70e94cc0fe14efb588cc99a3f466
Author: David Ames <email address hidden>
Date: Wed Mar 25 11:23:38 2020 -0700

    Enable update_peers method on cluster.available

    Closes-Bug: #1858132
    (cherry picked from commit 720a5922db534661c26c37f9bc136f206f179cde)

    Change-Id: I5716c28dd45b2d33325cf09487107543b8a42626

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715567
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=d4af1718e7d7acbf3f5972152d263a28e6e47058
Submitter: Zuul
Branch: master

commit d4af1718e7d7acbf3f5972152d263a28e6e47058
Author: David Ames <email address hidden>
Date: Fri Mar 27 15:01:30 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: I559e45ded1268f200fffdba8a27f6fc1d6b5db25
    Closes-Bug: #1858132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715560
Committed: https://git.openstack.org/cgit/openstack/charm-barbican/commit/?id=6d6cb5040b6ca6857ba18a3ff45e5c0770b4b2b1
Submitter: Zuul
Branch: master

commit 6d6cb5040b6ca6857ba18a3ff45e5c0770b4b2b1
Author: David Ames <email address hidden>
Date: Fri Mar 27 14:48:07 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Icf242fefbae99682aae2e666c246ac94edeb3f39
    Closes-Bug: #1858132

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716640

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716641

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716643

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716645

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716646

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716647

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716648

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716649

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716641
Committed: https://git.openstack.org/cgit/openstack/charm-placement/commit/?id=8a254c07e9cbd51c7c9af6a7c57290ce2000b33d
Submitter: Zuul
Branch: stable/20.02

commit 8a254c07e9cbd51c7c9af6a7c57290ce2000b33d
Author: David Ames <email address hidden>
Date: Fri Mar 27 15:02:42 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Iaa3fc3f2401981f6f0a2130a277a6da7439cff52
    Closes-Bug: #1858132
    (cherry picked from commit b4166e949cd9154975a1c526120c677089a204db)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716643
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/commit/?id=71c25f08bda39a2238ba6f5a865335d17e3f3f5c
Submitter: Zuul
Branch: stable/20.02

commit 71c25f08bda39a2238ba6f5a865335d17e3f3f5c
Author: David Ames <email address hidden>
Date: Fri Mar 27 15:00:09 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Ibb9d67095c85f0486d24bec01c356c7e7bfb008a
    Closes-Bug: #1858132
    (cherry picked from commit ef7a12ea5460a954db2e8a78d3897494a97e8c0c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716648
Committed: https://git.openstack.org/cgit/openstack/charm-barbican/commit/?id=d5656c7d88072da84b376e40533e8a66174e38da
Submitter: Zuul
Branch: stable/20.02

commit d5656c7d88072da84b376e40533e8a66174e38da
Author: David Ames <email address hidden>
Date: Fri Mar 27 14:48:07 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Icf242fefbae99682aae2e666c246ac94edeb3f39
    Closes-Bug: #1858132
    (cherry picked from commit 6d6cb5040b6ca6857ba18a3ff45e5c0770b4b2b1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716647
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=476992e53b3a88cf7b0096602f3262e000ade876
Submitter: Zuul
Branch: stable/20.02

commit 476992e53b3a88cf7b0096602f3262e000ade876
Author: David Ames <email address hidden>
Date: Fri Mar 27 15:01:30 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: I559e45ded1268f200fffdba8a27f6fc1d6b5db25
    Closes-Bug: #1858132
    (cherry picked from commit d4af1718e7d7acbf3f5972152d263a28e6e47058)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716640
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=06d8be372cbdd8afde9a94d55519eee5fd845360
Submitter: Zuul
Branch: stable/20.02

commit 06d8be372cbdd8afde9a94d55519eee5fd845360
Author: David Ames <email address hidden>
Date: Tue Mar 24 23:10:42 2020 +0000

    Fix HAProxy back ends

    Requires changes to charms.openstack and layer-openstack-api.

    Closes-Bug: #1858132
    Depends-On: I5716c28dd45b2d33325cf09487107543b8a42626
    (cherry picked from commit d6622db11d400d4321739bcddc7870ee4f2879d0)

    Rebuild also resolves bugs relating to certificate verification
    for internal and admin endpoints.

    Closes-Bug: #1866741

    Change-Id: Iaf20a0cd7bd2bc9accb20bd8987ed87289059e1c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716646
Committed: https://git.openstack.org/cgit/openstack/charm-manila/commit/?id=8b44055848e4c72e7cb7d0bdfc4d10c40a662601
Submitter: Zuul
Branch: stable/20.02

commit 8b44055848e4c72e7cb7d0bdfc4d10c40a662601
Author: David Ames <email address hidden>
Date: Fri Mar 27 14:57:18 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: Ic9b25431c1f5574bad855731d0210a9dbedacfc7
    Closes-Bug: #1858132
    (cherry picked from commit c1c2278530f2b852bbbb1f1df156225d0df5f3a2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716645
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=0622749212e4d74ca935e15b6dabf1e00700afc5
Submitter: Zuul
Branch: stable/20.02

commit 0622749212e4d74ca935e15b6dabf1e00700afc5
Author: David Ames <email address hidden>
Date: Fri Mar 27 14:43:15 2020 -0700

    Update peers with API addresses

    Use default_update_peers which will publish admin, internal and public
    addresses to peers.

    Change-Id: I1bf1f7272ab42c248ae86fede17c5cdd5f3892bc
    Closes-Bug: #1858132
    (cherry picked from commit 38353df4d2f1120bcbf383a2e6c537004ccc52e7)

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Ames (<email address hidden>) on branch: stable/20.02
Review: https://review.opendev.org/716649
Reason: Stable charm rebuilt for a different change.