haproxy stats http configuration is suboptimal

[Michele Baldessari]

a) The haproxy.stats stanza in haproxy config file has pretty much remained the same since newton:
listen haproxy.stats
  bind 192.168.24.8:1993 transparent
  mode http
  stats enable
  stats uri /
  stats auth admin:tRJre6PnQuN4ZwqKYUygTJArB

(what has changed post osp10 is that we also enabled the unix local domain stat socket for haproxy):
stats socket /var/lib/haproxy/stats mode 600 level user
stats timeout 2m
user haproxy

b) what we do today with the haproxy stats makes little sense:
- we bind it to the VIP running on the control-plane network on all controller nodes
- de facto we allow to look at the haproxy stat info via web only on the node holding the ctlplane VIP
- since haproxy does not share stats across nodes, we're effectively limited at looking at the stats info on a single node.

Now imagine ctrl-0 holding the internal_api VIP and ctrl-1 holding the ctlplane VIP. Basically now the only stats you will be able to see are the ones relative to keystone_admin (which for other silly reasons has been moved to ctlplane by default) and very little else.

We need to fix this in two ways:
1) Make it so that an operator can customize the network on which haproxy stats listens on, as currently that is hard-coded to the ctlplane
2) Let it listen on the controller's IP on the defined network
3) Bonus points for not changing current configuration and have it listen on the ctlplane vip as well.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/659926
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=e76519d2c8324e71db1871e2a9219eb66d0ce5c4
Submitter: Zuul
Branch: master

commit e76519d2c8324e71db1871e2a9219eb66d0ce5c4
Author: Michele Baldessari <email address hidden>
Date: Fri May 24 14:28:02 2019 +0200

    Fix tripleo::haproxy::stats to be more correct and flexible

    In this change we do three things:
    1) We make the class parameter 'ip' also be a list so that multiple
       bind addresses are possible
    2) We remove the hard coded 1993 and move it to a parameter
    3) Instead of passing only the controller_virtual_ip as the only bind
       address on all controllers which makes no sense (see linked LP)
       we also bind to the IP specified in the hiera key
       'tripleo::haproxy::haproxy_stats_bind_address'.

    Tested this change with the accompanying THT patch and correctly
    got the haproxy stats on a custom network (internal_api and the
    controller vip):

    listen haproxy.stats
      bind fd00:fd00:fd00:2000::16:1993 transparent
      bind 192.168.24.15:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:password

    I did not remove the controller_virtual_ip binding as that might be
    a breaking change for operators. We could think about deprecating it
    and removing it eventually.

    Related-Bug: #1830334
    Change-Id: Iab5f11c3065ff34a3543621554e7f05161d069f2

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/662287

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662287
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ef577909d6b5b0db6ea829e2325aed989d91ba35
Submitter: Zuul
Branch: stable/stein

commit ef577909d6b5b0db6ea829e2325aed989d91ba35
Author: Michele Baldessari <email address hidden>
Date: Fri May 24 14:28:02 2019 +0200

    Fix tripleo::haproxy::stats to be more correct and flexible

    In this change we do three things:
    1) We make the class parameter 'ip' also be a list so that multiple
       bind addresses are possible
    2) We remove the hard coded 1993 and move it to a parameter
    3) Instead of passing only the controller_virtual_ip as the only bind
       address on all controllers which makes no sense (see linked LP)
       we also bind to the IP specified in the hiera key
       'tripleo::haproxy::haproxy_stats_bind_address'.

    Tested this change with the accompanying THT patch and correctly
    got the haproxy stats on a custom network (internal_api and the
    controller vip):

    listen haproxy.stats
      bind fd00:fd00:fd00:2000::16:1993 transparent
      bind 192.168.24.15:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:password

    I did not remove the controller_virtual_ip binding as that might be
    a breaking change for operators. We could think about deprecating it
    and removing it eventually.

    Related-Bug: #1830334
    Change-Id: Iab5f11c3065ff34a3543621554e7f05161d069f2
    (cherry picked from commit e76519d2c8324e71db1871e2a9219eb66d0ce5c4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/659925
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=45f5c283e32e4753360c4e4c9627043b20a30e24
Submitter: Zuul
Branch: master

commit 45f5c283e32e4753360c4e4c9627043b20a30e24
Author: Michele Baldessari <email address hidden>
Date: Sat May 18 21:18:48 2019 +0200

    Fix haproxy stats network binding

    a) The haproxy.stats stanza in haproxy config file has pretty much remained the same since newton:
    listen haproxy.stats
      bind 192.168.24.8:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:tRJre6PnQuN4ZwqKYUygTJArB

    b) what we do today with the haproxy stats makes little sense:
    - we bind it to the VIP running on the control-plane network on all controller nodes
    - de facto we allow to look at the haproxy stat info via web only on the node holding the ctlplane VIP
    - since haproxy does not share stats across nodes, we're effectively
      limited at looking at the stats info on a single node.

    Now imagine ctrl-0 holding the internal_api VIP and ctrl-1 holding the
    ctlplane VIP. Basically now the only stats you will be able to see are
    the ones relative to keystone_admin (which for other silly reasons has
    been moved to ctlplane by default) and very little else.

    Tested this and am able to bind the haproxy stat to another network
    and to have it listen to the IP of the node on said network (in addition
    to the ctrlplane vip which we do not remove as it might break stuff):

        listen haproxy.stats
          bind fd00:fd00:fd00:2000::16:1993 transparent
          bind 192.168.24.15:1993 transparent
          mode http
          stats enable
          stats uri /
          stats auth admin:password

    Closes-Bug: #1830334

    Depends-On: Iab5f11c3065ff34a3543621554e7f05161d069f2

    Change-Id: If2ee15f1e0fcf6d077cba524fad75dec7e1144b6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/662388

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662388
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8b3fceb0bebef5dedbbb18e235f1e85c24c69302
Submitter: Zuul
Branch: stable/stein

commit 8b3fceb0bebef5dedbbb18e235f1e85c24c69302
Author: Michele Baldessari <email address hidden>
Date: Sat May 18 21:18:48 2019 +0200

    Fix haproxy stats network binding

    a) The haproxy.stats stanza in haproxy config file has pretty much remained the same since newton:
    listen haproxy.stats
      bind 192.168.24.8:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:tRJre6PnQuN4ZwqKYUygTJArB

    b) what we do today with the haproxy stats makes little sense:
    - we bind it to the VIP running on the control-plane network on all controller nodes
    - de facto we allow to look at the haproxy stat info via web only on the node holding the ctlplane VIP
    - since haproxy does not share stats across nodes, we're effectively
      limited at looking at the stats info on a single node.

    Now imagine ctrl-0 holding the internal_api VIP and ctrl-1 holding the
    ctlplane VIP. Basically now the only stats you will be able to see are
    the ones relative to keystone_admin (which for other silly reasons has
    been moved to ctlplane by default) and very little else.

    Tested this and am able to bind the haproxy stat to another network
    and to have it listen to the IP of the node on said network (in addition
    to the ctrlplane vip which we do not remove as it might break stuff):

        listen haproxy.stats
          bind fd00:fd00:fd00:2000::16:1993 transparent
          bind 192.168.24.15:1993 transparent
          mode http
          stats enable
          stats uri /
          stats auth admin:password

    Closes-Bug: #1830334

    Depends-On: Iab5f11c3065ff34a3543621554e7f05161d069f2

    Change-Id: If2ee15f1e0fcf6d077cba524fad75dec7e1144b6
    (cherry picked from commit 45f5c283e32e4753360c4e4c9627043b20a30e24)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/662494

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/662497

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662494
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=73fdcc0d62c74deee8a67cd28364cc5e4b42dea8
Submitter: Zuul
Branch: stable/rocky

commit 73fdcc0d62c74deee8a67cd28364cc5e4b42dea8
Author: Michele Baldessari <email address hidden>
Date: Fri May 24 14:28:02 2019 +0200

    Fix tripleo::haproxy::stats to be more correct and flexible

    In this change we do three things:
    1) We make the class parameter 'ip' also be a list so that multiple
       bind addresses are possible
    2) We remove the hard coded 1993 and move it to a parameter
    3) Instead of passing only the controller_virtual_ip as the only bind
       address on all controllers which makes no sense (see linked LP)
       we also bind to the IP specified in the hiera key
       'tripleo::haproxy::haproxy_stats_bind_address'.

    Tested this change with the accompanying THT patch and correctly
    got the haproxy stats on a custom network (internal_api and the
    controller vip):

    listen haproxy.stats
      bind fd00:fd00:fd00:2000::16:1993 transparent
      bind 192.168.24.15:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:password

    I did not remove the controller_virtual_ip binding as that might be
    a breaking change for operators. We could think about deprecating it
    and removing it eventually.

    Related-Bug: #1830334
    Change-Id: Iab5f11c3065ff34a3543621554e7f05161d069f2
    (cherry picked from commit e76519d2c8324e71db1871e2a9219eb66d0ce5c4)
    (cherry picked from commit ef577909d6b5b0db6ea829e2325aed989d91ba35)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662497
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b56035e655caddb26d6f61b3616531114f401153
Submitter: Zuul
Branch: stable/rocky

commit b56035e655caddb26d6f61b3616531114f401153
Author: Michele Baldessari <email address hidden>
Date: Sat May 18 21:18:48 2019 +0200

    Fix haproxy stats network binding

    a) The haproxy.stats stanza in haproxy config file has pretty much remained the same since newton:
    listen haproxy.stats
      bind 192.168.24.8:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:tRJre6PnQuN4ZwqKYUygTJArB

    b) what we do today with the haproxy stats makes little sense:
    - we bind it to the VIP running on the control-plane network on all controller nodes
    - de facto we allow to look at the haproxy stat info via web only on the node holding the ctlplane VIP
    - since haproxy does not share stats across nodes, we're effectively
      limited at looking at the stats info on a single node.

    Now imagine ctrl-0 holding the internal_api VIP and ctrl-1 holding the
    ctlplane VIP. Basically now the only stats you will be able to see are
    the ones relative to keystone_admin (which for other silly reasons has
    been moved to ctlplane by default) and very little else.

    Tested this and am able to bind the haproxy stat to another network
    and to have it listen to the IP of the node on said network (in addition
    to the ctrlplane vip which we do not remove as it might break stuff):

        listen haproxy.stats
          bind fd00:fd00:fd00:2000::16:1993 transparent
          bind 192.168.24.15:1993 transparent
          mode http
          stats enable
          stats uri /
          stats auth admin:password

    Closes-Bug: #1830334

    NB: Cherry-pick not 100% clean as it applies to
    puppet/services/haproxy.yaml and not docker/services/haproxy.yaml

    Depends-On: Iab5f11c3065ff34a3543621554e7f05161d069f2

    Change-Id: If2ee15f1e0fcf6d077cba524fad75dec7e1144b6
    (cherry picked from commit 45f5c283e32e4753360c4e4c9627043b20a30e24)
    (cherry picked from commit 8b3fceb0bebef5dedbbb18e235f1e85c24c69302)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/662930

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/662931

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662930
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=87744ef3290465e18fb832dfd20d2d2df197a55d
Submitter: Zuul
Branch: stable/queens

commit 87744ef3290465e18fb832dfd20d2d2df197a55d
Author: Michele Baldessari <email address hidden>
Date: Fri May 24 14:28:02 2019 +0200

    Fix tripleo::haproxy::stats to be more correct and flexible

    In this change we do three things:
    1) We make the class parameter 'ip' also be a list so that multiple
       bind addresses are possible
    2) We remove the hard coded 1993 and move it to a parameter
    3) Instead of passing only the controller_virtual_ip as the only bind
       address on all controllers which makes no sense (see linked LP)
       we also bind to the IP specified in the hiera key
       'tripleo::haproxy::haproxy_stats_bind_address'.

    Tested this change with the accompanying THT patch and correctly
    got the haproxy stats on a custom network (internal_api and the
    controller vip):

    listen haproxy.stats
      bind fd00:fd00:fd00:2000::16:1993 transparent
      bind 192.168.24.15:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:password

    I did not remove the controller_virtual_ip binding as that might be
    a breaking change for operators. We could think about deprecating it
    and removing it eventually.

    Related-Bug: #1830334
    Change-Id: Iab5f11c3065ff34a3543621554e7f05161d069f2
    (cherry picked from commit e76519d2c8324e71db1871e2a9219eb66d0ce5c4)
    (cherry picked from commit ef577909d6b5b0db6ea829e2325aed989d91ba35)
    (cherry picked from commit 73fdcc0d62c74deee8a67cd28364cc5e4b42dea8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662931
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=82875a493377ef16744660657b7381361595cbd1
Submitter: Zuul
Branch: stable/queens

commit 82875a493377ef16744660657b7381361595cbd1
Author: Michele Baldessari <email address hidden>
Date: Sat May 18 21:18:48 2019 +0200

    Fix haproxy stats network binding

    a) The haproxy.stats stanza in haproxy config file has pretty much remained the same since newton:
    listen haproxy.stats
      bind 192.168.24.8:1993 transparent
      mode http
      stats enable
      stats uri /
      stats auth admin:tRJre6PnQuN4ZwqKYUygTJArB

    b) what we do today with the haproxy stats makes little sense:
    - we bind it to the VIP running on the control-plane network on all controller nodes
    - de facto we allow to look at the haproxy stat info via web only on the node holding the ctlplane VIP
    - since haproxy does not share stats across nodes, we're effectively
      limited at looking at the stats info on a single node.

    Now imagine ctrl-0 holding the internal_api VIP and ctrl-1 holding the
    ctlplane VIP. Basically now the only stats you will be able to see are
    the ones relative to keystone_admin (which for other silly reasons has
    been moved to ctlplane by default) and very little else.

    Tested this and am able to bind the haproxy stat to another network
    and to have it listen to the IP of the node on said network (in addition
    to the ctrlplane vip which we do not remove as it might break stuff):

        listen haproxy.stats
          bind fd00:fd00:fd00:2000::16:1993 transparent
          bind 192.168.24.15:1993 transparent
          mode http
          stats enable
          stats uri /
          stats auth admin:password

    Closes-Bug: #1830334

    NB: Cherry-pick not 100% clean as it applies to
    puppet/services/haproxy.yaml and not docker/services/haproxy.yaml
    NB: Tiny conflict due to context for the queens backport

    Depends-On: Iab5f11c3065ff34a3543621554e7f05161d069f2

    Change-Id: If2ee15f1e0fcf6d077cba524fad75dec7e1144b6
    (cherry picked from commit 45f5c283e32e4753360c4e4c9627043b20a30e24)
    (cherry picked from commit 8b3fceb0bebef5dedbbb18e235f1e85c24c69302)
    (cherry picked from commit b56035e655caddb26d6f61b3616531114f401153)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.