Appendix B: Example production environment configuration in OpenStack-Ansible: clarification of infra load-balancers

[Marcin Dulak]

- [X] This is a doc addition request.

I've posted a question on http://lists.openstack.org/pipermail/openstack/2018-January/045964.html but it has not been answered yet, so I'm trying here.

The current behavior of internal_lb_vip_address/external_lb_vip_addresses is unclear from the documentation. I've found several posts and irc logs people asking for a clarification of the purpose and settings of that load balancers.

Taking https://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/app-config-prod.html as an example I thought that internal_lb_vip_address of 172.29.236.9 corresponds to the deployment host, but when running setup-infrastructure.yml I see haproxy are being configured on the infra nodes (.11, .12, .13) and listen on 172.29.236.9:8181.

Since there is no service listening on 8181 on the deployment host 172.29.236.9 I'getting the behavior described in https://ask.openstack.org/en/question/104307/openstack-ansible-pip-issues-while-installing-the-infrastructure/

At the bottom of https://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/app-config-prod.html there is a section mentioning haproxy_keepalived_external_vip_cidr: "1.2.3.4/32" but it's not clear to me how 1.2.3.4 is related to the setup performed by openstack-ansible using the openstack_user_config.yml file provided on this page.

Does openstack-ansible (17.0.0.0b2 on CentOS7) take care of those load balancers or do I need to configure them manually, how?

-----------------------------------
Release: 17.0.0.0b3.dev68 on 2018-01-19 22:04
SHA: eb67707b5ddabef8b4536324bc2c096f23f16d17
Source: https://git.openstack.org/cgit/openstack/openstack-ansible/tree/deploy-guide/source/app-config-prod.rst
URL: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-prod.html

[Jean-Philippe Evrard (jean-philippe-evrard)]

I am sorry your question on lists.openstack.org didn't get an answer. My filters search for [openstack-ansible] which didn't match in you email. I have to tweak that :)

Reading the rest now.

[Jean-Philippe Evrard (jean-philippe-evrard)]

We configure haproxy on the nodes appearing in haproxy_hosts. By default those hosts are located on our "infrastructure" nodes.

When you have multiple haproxy_hosts configured in your openstack_user_config, we'll automatically mark the need of configuring those haproxy nodes with keepalived.

However, you have to configure keepalived information, because we can't guess it.
That keepalived information will be used to determine which VIP will be used on which NIC.

Those VIPs can then be used in openstack-ansible, for example, by giving the dns name matching each VIP address in the openstack_user_config under the internal lb vip address / external lb vip address configuration keys. Internal and external LB VIP addresses should be different, and if possible (depends on your architecture configuration), it would be even better if they are on different NICs/networks.

These lb vip addresses IPs should be reserved (still in openstack_user_config), to make sure no container takes it.

When all of that is done, you can run all your playbooks.

Here the issue you see is probably caused by an error in your repo server. Make sure all of the above is fine, and re-run your playbooks. If everything is still wrong after the repo-install.yml playbook, please contact us.

If you want, you can also configure your own load balancers, are openstack-ansible could be simply using them.

I am not sure if that answers your question.
Could you please tell us how we can improve there?

[Marcin Dulak (marcin-dulak)]

Since it's still unclear to me, I'm attaching conf files.
Could you have a look?

[Jean-Philippe Evrard (jean-philippe-evrard)]

First, could you tell me what isn't clear? I will have a look at the files.

[Jean-Philippe Evrard (jean-philippe-evrard)]

In your case, you have haproxy_hosts that's equal to infrastructure hosts, which are 3 machines.
By default the haproxy playbook knows there are 3 machines, and call keepalived for help.

The problem is in your keepalived configuration:
haproxy_keepalived_external_vip_cidr: "172.29.236.9/32"
haproxy_keepalived_internal_vip_cidr: "172.29.236.0/22"
haproxy_keepalived_external_interface: eth1
haproxy_keepalived_internal_interface: br-mgmt

You can't bind an IP on a network.

On top of that, you have configured the following in your openstack_user_config.
  internal_lb_vip_address: 172.29.236.9
  external_lb_vip_address: dev-os1

Let's say:
- your external interface is eth1 and has the ip 172.29.250.2/24, with the dns name dev-os1.openstack.local
- your in cloud interface is br-mgmt, has the ip 172.29.236.9/22, and you don't care about the dns mame.

Then your configuration would be:

in openstack_user_config:
- internal_lb_vip_address: 172.29.236.9/22
- external_lb_vip_address: dev-os1.openstack.local

in your user_variables:
haproxy_keepalived_external_vip_cidr: "172.29.250.2/24"
haproxy_keepalived_internal_vip_cidr: "172.29.236.9/22"

Simple as that!

[Jean-Philippe Evrard (jean-philippe-evrard)]

This doesn't look like a bug, but more a support request, so i am marking this bug as invalid.

[Jean-Philippe Evrard (jean-philippe-evrard)]

Please don't hesitate to answer on this bug with what you think is unclear in the documentation, and we'll clarify it.

[Marcin Dulak (marcin-dulak)]

Thanks.

I still think there are inconsistencies in the docs that cause confusion depending where one starts to read the docs.
Please go through these points and verify if the documentation makes sense.

0.
A couple of lines describing the configuration and purpose of the haproxy setup performed by default by openstack-ansible would be useful, and maybe also a diagram that reflects the current configuration.

1.
https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-test.html
uses the IP address of the infra1 host
internal_lb_vip_address: 172.29.236.11
in addition to
external_lb_vip_address: 172.29.236.10
However the end of the page states:
"
For this environment, you are using the same IP address for the internal and external endpoints.
"

2.
All configurations
https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-prod.html
https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-pod.html
https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-prod-ceph.html
contain
haproxy_keepalived_internal_vip_cidr: "172.29.236.0/22"
while it should be
haproxy_keepalived_internal_vip_cidr: "172.29.236.9/22"

3.
The
haproxy_keepalived_external_vip_cidr: "1.2.3.4/32"
present in the three above links, needs to be changed to a more meaningful value, like your proposed
haproxy_keepalived_external_vip_cidr: "172.29.250.2/24" (or maybe better "172.29.250.9/22")
and then the 172.29.250.0/22 network mentioned somewhere on the top of the documents.
One should also state that
haproxy_keepalived_external_vip_cidr and haproxy_keepalived_internal_vip_cidr
must not be in use by other servers, and that 172.29.250.9 corresponds to openstack.example.com.
As a consequence "172.29.250.9/22" should be included in the used_ips: section of openstack_user_config.yml I guess?
~